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' PREFACE 

Thie* publication is one a series of technical evaluation reports 
issued by the Northwest Regional Educational Laboratory to document / 

evaluation lindings for seleci:ed products* The subject of this report / 

/ 

is Research VHlizing Probtm Solving (RUPS)^ an instructional system / 
developed in the Improving Teaching Competencies Program* / 

r 

This technical report presents the data collected about the impabt 
of the system on the classrooms of teachers trained in two RUBS workshops* 
The information is intended, to provide evidence related to the impact of 
RJJBS training on students* Although this information is primarily 

I 

summative in nature^ it should also help these who may be considering 
modifying the system to increase the liljcelihood of achieving Impact on 
students* ^ 

An institutional technical review has been conducted by Laboratory 
specialists external to the Program and qualified evaluation consultants 
external to the Laboratory have also reviewed this report* 



Lawrence D* Fish 
Executive Director 
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INTRODUCXION , j 

The overall goal of the Improving Teaching Competencies Program / 

(ITCP) of the Northwest Regional Educational Laboratory (NWREL) as 

^Ived In Its Resource Allocation and Management Plan (ITCP, 1974) Is^ 

"To develop Instructional systefms for training school personnel In / 

process jkllls which will promote student salf-understandlng^ self^ 

sufficiency and Indapandtncft* " The BAM^ further specifies that: 

These Instructional systeins will be competency 
based low cost and cvass diffusible for preservlce 
and Inservlce training in; ' (a)- supportive 
currlcular materials 'whlch^ncourage pupils to ^ 
be active learn^s; (b) ve^l behaviors that \ 
enable students to derlvf^ personally usable / 
meanings for what they learn; (c)' analyzing and 
Improving individual teaching styles and using | 
problem sovllng processes; (d) using basic 
Interpersonal skills; (e) providing for continuous i 
growth of teachers and contributing to Increased 
functional capabilities of- organizations. 

This. report focuses on the Research Utilizing Problem Solving (RVPS) 

instructional system, on^ of several Instructional systems developed by 

the Improving Teaching Comp.etencles Program In accordance with these 

goals. Based upon, summatlve evaluation data, this technical report 

provides Information and judgments for the benefit of potential users 

of the system^ 

\ 

Purpose of the Technical Report 

This document reports on the outcome phase of the process- employed 
In the development of RVPS^ The development of this Instructional sys*- 

tem was based on a model which divides the work flow Into five phases: 

P . 

Planning, Pilot, Interim, Field Test^^ and Outcome. Each phase consists 
of specified develo^ent^ evaluation^ and field relation activities 
which differ according to the pha^e under consideration. A brief 
description of the model Is provided In Appendix A* * 

^ , 10 1 



In the final or outcome phas«» It Is assumed that the development 
of the Instructional system has been completed^ and Interest is 
focused on a summative evaluation of the system^ s ability to produce 
not only specified short*term outcomes^ but also long-term outcomes^ 
Short-term outcomes include the categories of participant satisfaCction^ 
skill acquisition^ awareness^ ^nd Immediate performance change. Long- 
term outcomes include retention of knowledge and information^ and the 
general Impact of the instructional^ system on secondary audiences such 
as students and/or peers. In this sense » tlie terms ^Imp^t study** and 
'^outcome evaluation** are used interchangeably throughout this report. 

The purpose' of this technical report^ then» is to pr^^sent the 
results of the outcome evaluation/impact study. Although the evaluation 
was conducted primarily fon: the purpose of assessing the long-term 
outcomes of FU?it some short-term outcomes are also addressed* The 
technical report of the outcome evaluation of 'the EVPS instructional 
system will rulf ill the contractual agreements of the Improving Teaching 
Competenciei: Program of NWREL with its funding agency^ particularly 
those agreements related to the development and evaluation of RUPS* 

Audiences for the Technical Report 

Several potential audiences have been considered in the planning 
and implementation of the outcome evaluation of the EUPS system and in 
the preparation of this report. It is anticipated that the information 
contained in this evaluation report will be relevant to the concerns 
of three groups: 

1* Personnel in the Improving Teaching Competencies Program at 
at NWREL who are responsible for possible revisions or 
extensions of the systems 

I 

2 
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2, Educators who may potentially use the system and who need 
valid. an,d reliable InConoatlon In order to choose among 
' ' Inservlce educational alternatives. These educators niay 
Include the teachers who deslte to become workshop partlci* 
pants, curriculum specialists, 'or those who provide training 
opportunities Cor teachers 

3« Members of the National Institute o£ Education OHIH) who 
monitor the progress and assess the quality of output from 
the Improving Teaching Competencies Program 

Report Format - s 

\ 

This report Is organized Into fou^ chapters. The first chapter 
described the BUPS actlon^research model, as well %s the hlstory^i 
objectives, and components o£ this Instructional system* The second i 
chapter briefly outlines the^ Impact^ study/outcoqie Bvaluatlon; tu^ 
evaluation procedures, jmethodology. Instruments and data analyses are 
described* and the relevant evaluation questions are presented* The - 
third chapter presents and discusses the\ results oi the Impact study* 
The f^rth chapter presents a summary of bindings and results. 





CHAP TER ONE: DESCRIPTION OF THE flt/PS INSTRUCTIONAL SYSTEM 

Rationale for the Development of the SUPS System 

In 1965 Miles identified ^'problem^solving adequacy" as a key 
variable for '^organizational health/' In 1967 the Flint Youth Study 
(Lippitt) of the University of Michigan cited action research procedures 

emphasizing an objective data ba$e for decision making as a primary 

ft ' ' 

need for school district personnel* In 1971 Cross reported a study at 
the AERA annual meeting which showed that principal's decision*making 
patterns are reactive^ rapid and probably^ strongly influenced by 
^Subordinates* f - ' ^" 

Studies such as , these were taken as evidence of the need for the ^ 
development of a systematic approach to problem solving, as a means of 

encouraging improvemenr in schools. Wliile numerous new teaching 

* ** 

strategies ^and materiails have%een introduced into/the schools^ they 

' " " . ' a- " ' 

have had the effact of allowing teachers more freedom and perhaps 

providing more pressure to attend to the functions of analyzings 

' '* * 

diagnosing and Identifying generail problem areas as well as indivit^ual 

'learner problems* However^ the studies mentioned above indicated that 

'\ ' ' ' ' . 

^re structured work was needed in these difficult functional areas. 

These studies al$o implied that educators needed instruction not only 

in developing skills of analysis^ diagnosis^ and identification of 

problem areas, but also In integrating and applying their knowledge to 

initiate improvements in their education'al environments. 

"Based on this inf orTnatidn* the developers of Research Utilizing 

V 

Problem Solvzng (RUPS) concluded that an instructional system was 
needed which would, deal with strategies for relevant problem solving 
.^and could be readily adapted to the needs o£ both teachers and students. 



The RVPS Instructional system was thus developed to aid the improvement 
of the problem solving capacity of the schools and school personnel. 
It was the Intent of th^ developers that the problem solving procedures 
RUPS offers would contribute to the improvement of the general quality 
of education as well as provide a means for Increased Individualization 
and other forms of Instructional Improvement. 

/ 

Ihe RVPS Process * 

RUPS^^itAs designed as a low^ost mass-dlffuslble instructional 
system Intended to Improve the problem solving effectiveness of educators 
and people working in related capacities. The target population for RUPS 
is primarily teachers' and administrators. However » school board members » 
pa rap ro^f ess iQ aaXsL»^pro lessor s ^ and- state department" of educationfl^^ ^— ~ 
^ personnel may also derive some benefit fvpm RUPS. Generally^ RUPS^is 
intended to b$ of use to any members of the above groups who have a 
problem they are working on» or a problem which is of concern to them; 

An action-research model forms the basis o£ RVPS. The RVPS instruc- 
tional system was designed to help educators plan and manage learning 
experiences more efficiently by providing training in act ion ^research 
procedures. The procedures involve learning and using skills in applying 
objective data and research findings to solve local problems and ^o 
achieve improvements in schools and classrooats. 

The RVPS action^research process is^ intended to be applicable when 
the problem solver senses a problem^ but 1*5 unclear about the kind of 
outcome that is required or desired. The process is based on a 5-step 
method of problem solving. These steps are; 

1. Identify a concern 

2. ' Diagnose the situation 

6 . \ 
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3* Consider action alternatives ' 
4« Test selected alternatives 
.5/ Adopt and diffuse 

The first step involves applying available knowledge and consists 
of identifying the problem in tenos of who or what causes it and who 
or what is affecJted by it* Step two demands a thorough diagnosis of 
the problem situation* The various forces that work for or against 
perpetuation of the problem are considered in this step* Once the 
relevant forces have been identified, in step three action alternatives 
are formulated'* This is accomplished through brainstorming — a kind of 
avalanche*style verbalization of ideals* Brainstorming is followed by 
r feasiMil ty- t-ee t-ing;^ — During-s teir~f o ur"t™b raltra tomed^'d^^5nari&'~exa'mi"ne^ 
to determine which one(s) might work best* The final step include the 
adoption and diffusion of the altemativeCs) identified during brain* 
Storming* A possible result of these actions may be the generation and 
identification of new knowledge and concem(s) which i'n turn may be 
used to identify other problems* Thus^, the entire RUPS process is 
cyclical*' ROPS is intended to be a problems-solving process that not 
only offers logical solutions to probl^ems but is also effective in 
uncovering problems that may, in a'-'sense, be camouflaged by related 
concerns* 

Goals and Objectives of the jK/Pg SYstem 

;The previous section briefly described the action-research process 

* 

which forms the basis of the i?£/P£^ instructional system* The overall 

goal, of RUPS is; 

The Improvement of educational systems by using problem 
solving processes to identify problems and generate and 
effect solutions that are relevant to the needs of both 
local school systems and individuals 





/ 

* ■ 


i 
1 

'\ 
J 

• 




To accomplish this end» i?£/P£^ has two main objectives;^ 




> 


^ To Introduce and Instruct participants In the use of 
the RUPS process. >s a ^eans of working toward improve- 
ments In their- acbool setting 


• 




To provide part^lpants with opportunities ta. 
experiment^ with and experlen^ce ways of Increasing 
te'amwor-k skills that can facilitate problem solving 
* In school environments 


- 

• 




The developers of RUPS Identified a number of activities which 






focus on' the skills and behaviors expected to be demonstrated by 






educators participating In RUPS training* Since these actlvltlls form 


• 




the* basis of the 5*step RUPS process^ performance of these activities 






Is necessary for the accomplishment of the ^o^ls and objectives of the 






system* The activities have thus £een organized by the evaluators and 

» -J- 

— devfrlopers-of -the -system to correspond to the-two-objectlves-scated 


• 


3 

'^above* This presentation Is not Intended to be prescriptive nor are 






the two categories mutually exclusive* Rather^ the activities can be 






loosely catj^gorlzed In terms of: (a) those that emphasize the develop- 






ment of specific tasli^orlented skills and behaviors r^la^ted to the 






actlon-^research techniques and procedures » arid (b) those that emphasize 


V 


1 


the development of Interpersonal (teamwork) skills and behaviors* The 






activities that comprise the RUPS process are; 






1* Task "orientation skills: 






Applying the four guideline criteria for writing a problem 
statement 






^Previous development and evaluation documents have Identified other 
statements as the^objectlves^ goals and purposes of the RUPS system* 
It Is the opinion of the authors that these differences are basically . 
semantic and no real difference of opinion as to the goals of this 
system exists* The authors feel» however^ that the organization and ' 
presentation of goals .and objectives in this section offers a more 
coherent and logically consistent picture of the Intent of the WPS 
instructional system' than given earlier* 


• 
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Using force-field diagnostic' techniques 

Selecting and creating Instruments for data gathering 

Spotting and analyzing major results of data collected 

Using criteria for deriving implications from research 
findings 

Applying guidelines for planning and Implementing action 
alternatives 

Evaluating solution plans 

Planning a backhome project 

Conducting a backhome RUPS project * 

Brainstorming action alternatives to meet Implications 
derived from findings 

2. , Interpersonal (teamwork) skills: 

Paraphrasing Interpersonal communications 

Diagnosing teamwork relations 

Identifying one*5 personal style of operatlonallzlng 
dimensions of teamwork behaviors 

Utilizing concepts and skills of giving and receiving 
feedback 

Using the RUPS or actlor 'research process, teachers and students 
can adapt the above activities to promote imprbvements In classrooms 
and schools more ef f e%^tlvely* 

Content of the RUPS System — A Brief Description 

The RUPS Instructional system was desl^gned to provide training 
for educators In skills of using objective data and research findings 
as they apply a process of action-research tq^ achieve local Improve- 
ments. This training Is accomplished by guiding partlclpc^nts through 
■ A * 

a structured sequence c^f subsets or units, each consisting of a series 
of concept papers, group discussions, and exercises* These units are 
designed with the Intent of developing the participants' knowledge 

17 
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and abilities to apply this process to the Identification and diagnosis 
of classroom^ school or peer-related problems. 

The Instructional strategy of the RUPS systeui Is based on a pattern 
of repeated objective diagnosis carried out in small training groups of 
three or six pi^rsons* The knowledge gained through this process provides 
the basis for the selection and design of action plans to splve the 
identified problaiis* A simulation exercise provides opportunities for 
participants to practice skills in training groups and learn to observe 
and improve their actual teamwork behaviors in the workshop setting* 

J 

In other words » participants train each other using criteria provided 

in the materials* Basic Interpersonal and group teamwork skills are 

stressed in relation to the participants* efforts to apply the RUFS 

problem solving process to the simulation problem* 

In the simulation^ the workshop participants learn to U3e the RUPS 

mo^el as they; r 

Helg Mrs* Jones analyze her classroom situation in which the 
students apparently do not want to learn 

-Select data gathering, instruments ^nd process results of data 
gathered to redia^^nose the problem 

Create a plan to enable her children to become more active learners 
The **>Irs* Jones** simulation 1$ a primary training tool for the 
presentation of the RUPS process* The simulation and the acdompanylng 
activities and discussions provide opportunities for RUPS participants 
to familiarize thauselves with the 5-step process » gain experience 
and understanding in the use of ^specific actionnresearch procedures and 
techniques^ and share their knowledge with their training group. 

Once a number of subsets are completed paiftlclpants begin to 
concentrate on their oxm real problem^ by planning a RUPS project to 

10 
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be Implemented In their own schools^ thus working toward whatever goals 
are relevant to them personally. 

Although the vehicle for communicating information and directing 
activities remains relatively stable in the RUPS system (i*e*» the use 
of concept papers^ structured activities related to the simulation^ and 
grou^ discussions to convey information), each subset differs from the 
others in terms of both its specific objective(s) and content* For 
further information^ refer to the RUPS leader* s manual as well as previous 
development and evaluation plans (Butman» Jung and Rothlind» 1972)* 

Product Desi_gn 

A total of 33^ training hours is required to complete the RUPS 



instructional system* training is conducted in a worEshi^ setting* 
Th€^ instructional strategy calls for one 27H-hour workshop and two 
three^hour followup sessions held six and twelve weeks subsequent to 
traVing* ' 

I * 
The RUPS instructional system is composed of Id sequential and 

cumulative subsets of training units. The participant's manual is 
arranged in accordance with this sequence* The developers of RUPS warn 
against identifying and isolating specific activities and concepts from 
the system since it should result in the loss of certain features built 
into the systan for the expressed purpose of reinforcing previously 
learned knowledge and effecting an integration of that knowledge with 
ne^ knowledge* Foe this reason^ the system*s developers emphasize its 
cumulative and sequential nature and caution against any deviation from 
the prescribed instructional strategy. ^ 

The time allotted for the completion of each unit vaties; the 

\ 

average time is approximately two hours per uAit* Each training unit 

11 
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begins with a statement of the subset^s purpose and objectives. There 
Is then an agenda or list of steps the participants will go through to 
complete the unit. The Infonoatlon In each subset Is conveyed through 
the use of concept papers^ activities^ and group exercises. 

At the beginning of the workshop^ the leader distributes a set' of 
materials to each participant. The materials Include a booklet on data 
gathering Instruments (Fox» Luszkl and Schmuck^ 1966) and the training 
manual which contains all the handouts with the exception of those 
concerned with the simulation* These latter handouts are renoved from 
each manual and are distributed by the leader at the appropriate times 
Indicated In the lea^ier's Instructional strategy* (The leader*s manual 
differs from the trainees* manual dtily^ln that the leader* s manual 
Includes an Instructional strategy for each step.) 

The Instructional strategy Is very specific concerning the leader's 

role In the training. The workshop leader Is responsible for giving 

directions passing out materials^ presenting charts^ reviewing the. 

agen^ of .each subset and clarifying Instructions* The leader operates 

a tape recorder In the early .training unlts^ times the exercises and 

occasionally leads a group discussion concerning the progress of the 

workshop* The design of the Instructional system does not call Cor the 

workshop, leader to be an expert In either the WJ^S process or teamwork * 

skills* The Instructional strategy does not Include any activities 

* 

where the leader directly Instructs the participants In the skills they 

are learning* Instruction Is provlde^d by the Interaction of the 

participants with, each other In the exercises* 

The Instructional strategy .for each training unit Is In the 

leader* s manual* It Includes a listing of the steps and the materials 

needed for each» the approximate amount of time each step should 'take 
12 
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to be completed^ and detailed directions for the participants. To aid 

the leader's preparation, there Is additional Information concerning 

the specific Intent of each activity and steps ulthln each subset. 

This Includes a statement of purpose for the subset, the objectives, 

the rationale for the inclusion of '*each step as It Is found In the 

design, and a list of all materials, for the subset* 

Continuous active participation is effected during the workshop 

through the use of the simulation described earlier- The simulation 

J 

Is geared primarily to teachers at the elementary school level and 
experience* The entire design emphasizes participant practice with 
skills of the 5-step RUPS problem solving process. 

Developmental History of j?i/Pg 

The original prototype for RVPS was created over a 3--year* period 
between 1966 and 1969* Many of the basic Concepts In Ii}JPS were derived 
from earlier work of Kurt Lewln which were then transmitted by 
Ronald Llppltt to the senior developer, Charles Jung* Mark Chesler 
and Robert. Fox at the Center for Research on the Utilization of 
Scientific Knowledge of the Institute for Social Research, University 
of Mlchlgan^^ provided important contributions to the development of the 
Instructional system by exploring the assumptions of thfs training 
design* Much of the Interpersonal content of the system and generallza-^ 

tlons underlying the design were derived from the work of the Cooperative 

'* 

Project for Educational DeV^elopment (COPED) of the National Training 
Laboratories* 

The first attempt to implement a complete design of the RUPS 

Instructional system was made In collaboration with the Natlon^tl Bo^d 

* * 

of Education of the Methodist Church* The next opportunity to further 
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develop the system came In connection with the COPED project. Funding 
was provided by the Research Training Branch of the United States Office 
ofs Education/ Training during this stage was conducted with the Brooklyn, 

K 

Jackson, Levonia, and Detroit public school districts of Michigan* 
In I968t the National Association of Classroom Teachers (ACT) 
supported the testing of the Instructional system at three successive 
annual conventions. It was at this time that the ''Mrs. Jones" Simula*^ 
tlon was 'Introduced Into the training design* An attempt also was made ''^ 
to further explicate the system through an analysis of the objectives 
of the training. This study was conducted by Ren£ Pino under the 
supervision of Robert and Elizabeth Corrlgan* The Corrlgans conducted 

..a-major field evaluaMon- of— EVPS' 3ls part^of— an-E&EA^llrle"~iTr"project 

In the Atascadero, California, public schools. 

' On the basis of their experiencefi with the M/PS Instructional systems 
in the cont-exts described above* the developers decided to make several 
revisions In the training materials. Several of the concept papers 
were rewritten, the role^of group dynamics In the training was emphasized, 
and the section on the backhome ^lmpx;ovenent project was added* 

The final revisions of the materials were accomplished In 1970 by a 
cooperative venture' Involving Northwest Regional Educational Laboratory, 
National Education Association, Oregon Education As^dclstlon, Washington 
Education Association and Central Washington State College. The resulting 

V 

product was entitled Reseavph Utilizing Problem Solving: Cla$8V00m 
Version* A companion edition for administrators ^was entitled Reeeardh 
Utilizing Problem Solving: Administrator's Vereion.^ A field test for 
this RUPS version was conducted In 1971* These results have been reported 
In Teohnioal Report No. ^?: Researoh Utilizing Problem Solving (Butman^ 
Jung and Rothllnd, 1972). 
14 
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CHAPTER TWO: "DESIGN OF THE IMPACT STUDY 

The purpose of this chapter Is to summarize the evalutlon design 
for the EVPS outcome evaluation/impact study* Implementation decisions 
concerning the design of the RUPS Impact study were Influenced by a" 
variety of situational constraints* As a result^ the evaluation design 
as It was Implemented differs from the design proposed for this Impact 
study* 

There are four sections In this chapter* The major evaluation 
questions are given In Section I* Section II discusses the Instruments 
used In this study* The sub|.ects, of the study are described In 

Section lilt and the procedures t^s^d In carrying^ out the Impact study 

" — ^ " ' ' ■" ^* 

will be discussed In Section IV* 

Evaluation Questions / 

'I ^ 

This sectlj)n lists the major long-term and short-term outcome 

^ ^> 

questions to be addressed In this report* The focus of theare questions 
Is on partlcl^nt satisfaction^ use and subsequent effects of the 
knowledge and skills learned by participants In RUPS rather than on' the 
mere acquisition of knowledge and skills* The questions have evolved 
from discussions with program developers about their Intent for the 
RUPS Instructional system and from analysis by program evaluators of 
what would provide rigorous^ yet reasonable^ te«tts of the system. 

It Is assumed that all of these questions^ which have been sorted 
Into two categories In Table 1» will be of Interest to both developers 
and potential users of the Instructional system* "The sources of 
Information and the data collection methods are identified In Table 1; 
on the right side of tKe page the procedures for analyzing and reporting 
Information for each evaluation question are designated* 



Table 1 



Evaluation Questions for the RUPS Impact Study 



Qu««clon« 


Inf ormadon 


Dae a 

Colloctlon 
Hochodo 


AiMlyolo ond 
Kopocdng 

FTocOdUTOO 


Qu^ticn^ ii^lcted to tht '^uatiZif and 
Acaeptabilittj cf tha Cpnt^nt, Str<itegi4%, ^ 
and t-Sa^^trtaZ^ fci* thi In^tpuctionat Stf%um 








« 

Do cciliiM* c«p6cc ucl«f«ctlon wlch ch^ 
RUPS ln«cnjccibn«l «y«c«ttt 


RUPS 
^ tcalo«oo 


" ^oicccolnlni 
Quoodonnalco 


Doocdpdvo 
Suvmacy 


■Do ch«M p«rc«pclon« p«r«l«c ovtc • 
3-«onch p«clodT 




Followup 
Quoodoniulco 


Doocdpdvo 
c*tooco 


Do ccilniM p«cc«lv« ch« ccilnlai itifJtUPS 
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RUPS 
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Fooccca In Inf 
Quoodonnalco 


Doacdpdvo 




1 

Do chooo pcocoduno pocoloc ovoc i 
3-9onch p^clodT '* 

] ' 




Followup 
Quoodonnalco 


DotcUpclvo 
SuflMCy 1 
c-tooco ^ 


Whoc ICO cho old* offACCi thmt cooulc fcoa 
cho uoo of cht syic«n? ^ 


Oboorvocoi 
Evalu cocOi 
Uaoci 


Infonul 
; OboorvadotiOi 
UO«Itly Lo| 


tSuMaty of 
Find In [0 


Dooo RUPS coapoco «4vitici|«ouoly co 
coBpoclflvo tfiyo'of niodtif cho fiiM 
oucc^wt obJocdvMT 


EKIC 


Llcococuca. 
Sootch 


SuBttacy of 

rLnOLngO 


^otiono HtaUd to loiftg*Rm$% Training 
5//<*<:to 








Acdcudlnoi £ff«cco9 








Do ccalnooo' poccopdono of cholc McStLni 
uft^cocondlni of cb« ucodol chongo ovoc 
•* 0 3*Aonth pododT , 


WPS 
trolaooa 


"Voaccrolnlnc 
ond Followup 
Quosdoniulcoo 


c*tooco 


Do BVPS ccolnooo ohanio Ln cholc 
odonC4Clono cowtcd pcobl«i oolvln^T 


WPS \^ 
TcolnMa 


Ptoblon 
Solving 
Odoncadon 
Quoodonntlco 


c-Tooco 


Do WPS ccolnooo bocoM poco oblo co 
ld«fidfy pioblM* chac nood occindoti 4nd 
90CO tflllloi CO occ«nd co pfoblos oolvUi 
In ■ oyocMAcU vay chut do coochoco tfho 
hivo noc cocolvod ccololni? ^ 


RvFS 

TCalMOVt 
CoapocUon 

Ccoup 


FCOOiOB 

Solvla$ ^ 

Orloncadon 

Quoodonnalco 


Foch 'Analyolo 


^ lotuvlocol EllOSCff 








Do c&)PS ccolnMO dlocuao choir pcobi» 
oolvlni ktHMrlodio and oStlllo In cholc 


RUPS ^ 
tcolnooo ^ 


U«akly Lot 


Doffcdpdva 
SuoMty 


Do RUPS ccoineoo uoo ch4lc pcoblon 
oolvini knoirlodfo and oStlllo In cholc 
local ochooiot 


, SUPS 

tTOlil«OS 


UMStfy Lo| 


Doocdpdvo 
Sumaacy 


Inpocc 00 S«€ondocy Gcoupot 








^ Doos WPS ccalnlTtf cosulc tn chanfoo In 

ClOOOCOOB cllAOCOT 
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Ttolnooj 


School 

Acdvldoi 

Quoodonnalco 


c*tosco 


Do RUPS ccaln«co lnvolV« ch4lc «cud«ncj 
In problem solving occtyicl«o oore dtan 
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School 
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The pattern of relationships among outcome measures Is addressed 
through the technique of path analysis (Spaeth^ 1975). Evaluating the 
impact of teacher training on students often requires the definition- of 
a sequence of outcome measures which allow the training activities to 
be linked with student outcomes* When there are well defined student 
objectives^ evaluating the impact of teacher training Is straightforward* 

If student objectives are missing^ as they were for RUPSf evalua* 
tors must use what Information Is available to Infer a sequence of 
training effects* Such an Inferred sequence of ef f ects ^bears a strong 
resemblance to model building* It follows that evaluating the evaluators 
model becomes a significant aspect of the study* 

Path analysis Is an analytic technique Useful In explicating the 
evaluators' assumptions and In empirically evaluating Its efficacy* 
It also has potential for formative evaluation as It may help Identify 
"weak links'* In the chain of effects expected to. result from training* 
It was for these reasons that path analysis was used In^^the present 
evaluation study* ' 

Instrumentation * 

The formats and purposes of the Instruments used for the collection 
of data will be discussed In this section* All 'questionnaires with the 
exception of the School Activities Questionnaire (SAQ) were administered 
and explained by the e^aluator observing the workshop* Sample copies of 
all Instruments can be found In Appendix B* 

Background Questionnaire * This Instrument was developed by 
evaluation staff of the Improving Teaching Competencies Program to 
assess various demographic and attltudlnal variables of participants* 
The variables under consideration were: age» sex» employment position^ 
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educational backgrouivl» previous workshop experience^ expectations for 
the workshops and reasons for attending the workshop. Responses were 
written directly on the questionnaire and were later processed by 
trained coders. 

Problem Solving Orientation Queatlonnalre (PSOQ) . This Instrument 

was designed as a paper *and*penc 11 self-report measure of the orlenta* . 

tlon of teachers toward problem solving* In the questionnaire 

respondents are presented with a variety of realistic problem situations 

and asked to estimate the probability of their behaving or reacting In a 

manner consistent with the view of problem solving advocated through 

RUPS training* Responses to Items are made 'on a 5*polnt multlple-'Cholce 

scale Identical to that In the following sample Item: ^ 

^ Suppose something has gone wrong In your school** 

. somethlnjt that affects everyone and has everyone 
upset. What 'are the chances that you would remain 
quiet and wait for others to analyze the problem? 

1* Almost none (less than^a 10 percent chance) 

2. Maybe (about a 25 percent chance) 

3. Possibly (about a '50 percenc chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

The Items from the PSOQwere categorized according tc two classlfl* 
cation schemes determined by the content of the situation presented In 
the Item stem* The first classification s^^heme divides the s^ .jatlons 
described in the PSpQ Into two categories: (a) situations dealing 
primarily with Interpersonal relationships » and (b) situations dealing 
primarily with task accomplishments * 

The second classification scheme also divides the situatl^ons 
described in the PSOQ into two categories: (a) situations related to 
classroom problem solving activities^ .and (b) situations related to 
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none la8S room problem solving activities* Scores for each set of Items 

were the sum of the responses to each Item within that category* 

Two developers of the RJJPS Instructional systan and four evaluators 

who had never received BJVS training wef^e Instrumental In assigning the 

Items .to the categoi;J.es described above. Each developer and evaluator 

received a complete PSOQ, with each Item printed' on an Index card* 

Th^y were Instructed to work Indepepdently and assign Items dealing 

with task accomplishment or Interpersonal relationships. Items which 

the eval tt at -e r s^n d d eve loper s coul d-not unanimously assign to these 

two^categorles were not scored. Of the thlrty^one Items In the P^QQ, 

* 

only two were eliminated because of lack of unanimity. Twelve Items 
were classified as' tasks and seventeen as Interpersonal relationships* 

The same procedure of sorting cards was followed for the classroom 
and nonclassroom ^categories'* -Twelve Items wer^e placed In the classroom 
-sltuctlon category, seventeen Items were placed In the nonclassroom 
situation category, and Items were deleted duetto lack of unanimity* 

Data from a sample of 87 teachers who responded to the PSOQ were 
used to calculate split-half reliabilities, which were corrected with 
,the Spea^:^^n Brown Propfiecy formula* These 87 teachers were either 
participants In RV?S^ participants In two other NUREL workshops, or 
members of the conjbrol group used In the present study* Their pretest 
data were used to calculate the split half reliabilities* 

In addition the split half reliabilities,, test-retest reliabili- 
ties were computed with the same group of S7 teachers* Since the test- 
retest reliabilities were based upon pretest and posttest administrations 
designed to assess training effects, the test*retest reliabilities should 
be Interpreted as conservative estimates of stability. Table 2 Includes 

both the split half and test-retest reliability estimates* 
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^ A Table 2 
Reliabilities of PSOQ Scales 



PSQQ Scale 


Split Half. 


Test-Re test 


Task Acjcompl Ishnent 


.71 


.61 


Interpersonal Relationships 


.37 


.44 

L 


Classroom Situation ^ 


.60 


.61 


Nonclassrooa Situation 


.38 


.44 


Total Score * 




.59 



The task accomplishment and classroom situation scales were more 
/reliable than the^ interpersonal relationships scale or the nonclassroom 
situation sc.ale. The former two scales are more useful for evaluation 
purposes. The test-retest reliability for the total score on the PSOQ 
shows moderate stability for an instrument of this type. 

Since items appeared in more than one scale across classification 
t schemes (but not within classification schemes), it is only appropriat.e 
to examine scale intercorrelations within classification schemes. The 

Pearson product moment correlatidh between the task accomplidhment and 

-"ji 

interpersonal relationships scale was .55 for the pretest data and .54 
for £he posttest data. The classroom situation scale and the nonclass* 
room situation scale were inter correlated -32 on the pretest and -34 for 
the posttest. These correlations are statistically significant at the 
.05 level or greater. It is noteworthy that the pretest intercorrela- 
tions and posttest intercorrelations between the two scales in each 
classification scheme were almost identical. Further- information on the 
psychometric properties of the PSOQ is presented in Appendix C. 

School Activities Questionnaire (SAQ) . The SAQ Is a multiple-choic 
questionnaire designed to measure various aspects of classroom climate. 
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The SAQ used in this evaluation study was based upon the Student 



Activities Questionnaire, an instrument under development by the 



Institute for Behavioral Research in Creativity, (Ellison, CaUner and 
Fox, 1972). It. was compoBed of four of the original eight subscales* 
Four BubBcales were chosen as being moBt sensitive to the problem Bolying 
behaviors of the teacher, based on information of how past trainees had 
used the training in their classroom. The subscales were; (a) reinforce- 
liient of self-concept, (b) individualization of instruction, (c) democratic 
classroom control, and classroom participation* ' ~ / 

The following is a description of the subscales and scores used 
in the SAQ* The items in each subscale were keyed on an a priori 
judgmental basis directed toward the critical concept behind, each sub* 



scale* The score for each subBcale is a total of all item scores Un 
that subscale* 

^ 1* Reinforcement of Self-Concept * This score measures the 



by the student, either through personal contact or structured 

class activities* It i^s also a measure of the student's feelings . ^ 



student^s perception of the amount of positive feedback received 



about their work, offers encouragement f or' difflWlt taskSr-^and^ 
gives frequent positive feedback for good performances* ,JEhese^ 
students also report that they feel encouraged wheh they receive 




feedback and consequently develop pride in the£r school work. In 



addition to positive teachet contact, students also'report thac they 
have frequent class activities where, they have the opportunity to 



give each other feedback regatiding good work* 
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2* Individualization 'of Instruction * This score measures the 
extent that students perceive their teacher^ as sensitive to their 
own Individual needs » progress » and goals* High scores In this area 
reflect students who talk privately with their teachers .about them- 
selves and their ^school work and report that their teachers are 
sensitive to their strengths and weaknesses and the kinds of 
activities they enjoy* They further Indicate that the difficulty 
level of their classwork Is neither too hard nor too easy and that 
activities are most often Individualized to meet the needs and 
abilities of each student* 

3* Democratic Classroom Control . This score Is a measure of 
student Input Into classroom decision making^ planning of Individual 
activities and enforcement of rules. Students wltii high scores In 
this area report that they are allowed frequent Input through 
discussions and planning activities and that such decisions are 
made through the joint effort of teacher and student^* They .also 
Indicate that their teacher will permit a noisy classroom during 
many activities and stresses stud^ent participation rather than 
authoritarian control. 

4* Classroom Participation * This score measures student 
participation In classroom activities* The Individual Items In 
the score Involve frequency of classroom discussions^ number of 
students that typically participate^" and opportunities for parti- 
clpatlon In the classroom* High scores In this area Indicate 
frequent class discussions and activities where many students^ are 
called upon or given jthe opportunity to speak* They also Indicate 
that students have the opportunity to work with a variety of other 
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students on group^ projects and are often given the opportunity , 
to teach each other* 

5* Climate Composite * This score was an unweighted sum of 
the four scales of the SAQ*-^ Reliabilities for the four SAQ scales 
and the climate composite were calculated on data collected in the 
' present investigation by two techniques* The intraclass correla* 
tion (Haggard^ 1958) which is a simultaneous measure of within 
classroom agreement and. between classroom variability was used as 
the first measure of reliability* These reliabilities are shown 
In Table 3* The Intraclass correlation coefficient Is a function 



Table 3 
Reliabilities of SAQ Scales 







Intraclass Correlations 


SAQ Scale 


Test-Retest* 


pj-gb 


Post<^ 


Reinforcement of 
Self -Concept 


.28* 


.36** 


^,.27* 


Democratic Classroom 
Control 


.45** 


, .79** 


' .72** 


Individualization of 
Instruction 


.12 

* 


.21 


.33** 


Classroom Participation 


.59** 


.72** 


.58** 


Climate Compoaite 


.42** 


.57** 


.49**. 



^Number of classrooms ' 73^ number of students « 1213* 
dumber of classrooms « 84, number of students « 1499* 
^Number of classrooms ' 73, nunber of students « 1213* 
*p < *05* 
**p < *01* 

of within group and betwe^ group variation of student responses 
and constitutes a measure of consistency or reliability* As such^ 
it is a measure of reliability using the classroom as the unit of 
analysis* Intraclass correlations were computed for both the. 
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pratralnlng data and the posttralnlng data. The second reliability 
estimate used was a standard test-retest reliability* As Is true 
for the PSOQ reliabilities discussed above, these test-retest 
reliabilities must be considered as conservative estimates because 
of the Intervening treatment. SAQ reliabilities estimated are 
presented In Table 3* An Intercorr elation matrix Including the four 
scales and the climate comppslte from the SAQ and 13 other climate 
scales used In the evaluation of* Improving Teaching Competencies 
Program woi^ksjiops Is presented In Appendix D* 

Posttralnlng Questionnaire * The purpose of this questionnaire was 
to^ assess participants' perceptions of their learning experience* Il&ems 
were grouped to measure specific areas* These areas relaced to Xa) judg^ 
ments about the success of the workshop In achieving Its goals, (b) sel£-\. 
report of understanding of conceptual materials and acquisition of skills, 
and (c) feelings about the usefulness of the workshop for self and others* 

Part I of the questionnaire Is composed of statraents concerning 
the success of the woi^kshop In meeting Its goals* Participants 
responded on a 5-polnt rating scale ranging from 1 (*'not at all 
successful'^ to 5 ("completely successful")* 

.^art II consisted of a list of activities and skills* Participants 
vere asked to rate their "working understanding" of thos^ skills on a 
4*-polnt scale* The' scale range was; 



I don t undtrsttnd 1 don't undtrstand 1 understand the 1 understand the 

the concept and the concept but concept but could concept and could 

couXd not apply could apply the not apply the apply the akUXs 

the aklUs aklUa sklUa 

Questions concerning the satisfaction of the participants with the 
workshop and their perceptions of Its usefulness were posed In Fart HI. 
Participants responded or* a S-polnt scale* 
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The Probleat Statement * One of the activities of the SUPS workshop 
is the planning of a "backhome** i?£/P£^' project* Using the RUPS model, 
participants composed problem statements* These statements described 
the project or problems that participants anticipated undertaking in 
their local school settings* Statements were collected from all 
participants for use in this evaluation* 

The Weekly Log * The Weekly Log is an instrument of self-riport of 
problem solving activities* With this instrument each trainee indicated 
ve^^ly the number of timed he or she had engaged in various problem 
solving activities* The Weekly Log was composed of\ two forms* Form 1 
included ^ list of fourteen behaviors or activities! specif ied in SUPS 
training. Participants were asked to estimate the frequency with which^. 
they engaged in these problem solving activities during th\t preceding 
week* This form of the Weekly Log was intended to be completed and 

j \ * ^ ^ < 

returned in an- enclosed self-address6d, ^tamped envelope. every^Ftlday' j ^.^y lA--^ 
for nine weeks, October 25 through December 20. 

Form 2 of the Weekly Log was concerned with the nature of the 
problems the participants had been working on during the preceding 
tva weeks* Every problem worked on required an additional Form 2 
since a participant may have worked on several problems during a two- 
week period* This form also included questions concerning the progress 
> individuals had made on their problems and the difficulties or successes 
they had encountered* Form 2 was submitted together with Form 1 of the 
Weekly Log every other week* 

Followup Questionnaire * The Followup, Questionnaire was a modified 
version of the Posttraining Questionnaire* It covered the same three 
areas, but from the viewpoint of ?now that three months hav^ passed 
since you received SUPS training, "^ow do you feel about * * *" All 
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^uc one of che questions were mulclple-choice options; responses were 
recorded on an addlclonal opclcal scanning daca sheec* One open-ended 
question was asked concerning the participants' perceptions of the 
usefulness of the SUPS training* 

Subjects of the Study 

RUPS Treatment Group * The target population for the RUPS Instruc- 
tional system is primarily teachers, and recruitment efforts for this 
study focused on chls group* All fourth, fifth, and; sixth grade teachers 
from each of the two test areas were Invited to participate In che RUPS 
workshop* These teachers were offered free materials, training, and 
paid tuition for three hours of graduate credit In return for their 
cooperation In providing data and Information relevant to che evaluation,. 
Specifically, participants were asked to agree to; 

1* Attend every session In Ics entirety 
^' ' 2i : Participate In data collection during the workshop 

3* Allow approximately 30 minutes for the collection of 
Information from students In classrooms 

4* Keep a weekly log of problem solving activities for three 
months following the training 

Xc was originally anticipated that one major test site, Tacoma, 
Washington, would be used for the RUPS ^pact study* Due to a teachers' 
strike In mld^September, however, che Tacoma school district decided 
to withdraw' from the study* As a consequence. It became necessary to 
look elsewhere for subjects to participate In this study. 

As part of the general recruitment effort In Tacoma, recruitment 
procedures had already been Initiated In several school districts south 
of Tacoma. A number of subjects from these districts had signed up to 
participate In the RUPS workshop prior to che withdrawal of the Tacoma 
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school district* When the Tacoma districts withdrew from the study» the 
decision was made to use the subjects from the districts south of Tacoma 
and to recruit subjects from a' second major site In Seattle* 

Based upon considerations of the distance between the two test sites 
and the difficulties In cotmnutlng to and from the sltes^ arrangements were 
made to conduct workshops at both sites; one workshop was conducted In 
Tacoma for the participants from the districts south of Tacoma^ and a 
second workshop was conducted In Bellevue for the participants from the 
Seattle area* The combined results of^both workshops werd used In the 
analysis of data and Interpretation of the results* 

Comparison Group * The original evlauatlon design called for a 
^comparison group as well as an experimental group* The (Comparison group 
Was to have been randomly selected from those teachers who volunteered 
for RUPS training* The physical distance between ^h^ two workshop sites 
an^ the relatively small number of participants at each site mitigated 
against thls» however* The decision was made to provide RUPS training 
for/'^ll, participants at each site and explore alternative means of 
providing comparative data* ^ 

An evaluation study Involving two other experimental g^roups and a 
control group was scheduled to be conducted In the Seattle area two weeks 
before the RUPS workshop* Two groups were to receive training In the 
Interpersonal Influence (IHP) and the Group Process Skills (GPS) Instruc- 
tlonal systems^ which are two other Instructional systems developed by 
the Jjuprovlng Teaching Competencies Program of NHKEL* l]\e third group 
was a nontreatment control group which was to receive training In the 
i Interpersonal Jn/Zu^nc^Unstructlonal system the following Spring* 

\ 

The target populations for the GPS and INF Instructional systems 

. \ 

are also primarily teachers* Recruitment procedures for these groups 

" \ ' 27 



35 



\ 



were the same as those for the RUPS groups. Subjects for the GPS^JNF 

and nontreatment control group" were recruited from all fourth, fifth , 

and sixth grade teachers in the Seattle area* ?or these reasons teachers 

who were members of the nontreatment control group also served as the 

comparison group for the evaluation of RUPS* 

It was felt that comparison of test results from this group with 

the RUPS group would provide useful information, particularly in 

addressing questions directly related to the issue of Impact of training* 

* 

For this reason and for reasons of co^t and ava^ ^fth^^^^y ft^Hj^rt-c ^^ 
was 'considered feasible to use the nontreatment group as the comparison ^ 
group for ,the RUPS instructional system** 

Group- Sizes * The total number of participants in. both the treat- 
ment group and the comparison group is shown in Table 4* Table 4 also 
provides a breakdown of the total group into individual workshop groups* 



Table 4 

Treatment and Comparison Group Sites 



Treatment Group 




H 


Craparison Group 


N 


HUPS I Tacoma 




17 


Control 


24 


FUPSi Be^evue 




21 






>Total 




38 







Description of Treatment and Comparison Group * Both groups com* 
pleted a Background Questionnaire* Information collected included sex, 
age, occupation, years of teaching experience and education of the 
participant* Additional participant information concerning previous 
NWBEL workshop experience and reasons for attending the workshop was 
also obtained* The responses of both the comparison group ar^d the HUPS 
treatment group on all background variables are summarized in Table 5* 
28 
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Table 5 



Distribution of Participant Characteristics by Group 



* 


ROF$ TreatmenC 








Group 




ConCrol Group 




Per c en Cage 


N 


Pat a n ^ a a * 

c V 1 1^ V n b« ji B 


U 
St 


Sex: 










MaI« 


34*2 


13 


41 7 


1 n 

lU 


Feaule « 


63*1 


24 


JO * J 


14 


No rttpontc 


* 2*6 


1 


0 


0 


Ag«: 










Under 25 


2*6 


1 




1 
1 


25-3A 


47*4 


18 


41 * o 


1 fi 




18*4 


7 


1 ^ ^ 






Unr4 


— ^- 




— e_ 


55-64 


5*3 


2 


AV* 9 

Jl V * / 




64 and ovtr 


0 




0 


0 


No rttponst 


7.9 


3 






OccupACion: 










ElemcnCary School Teacher 


100*0 


38 


Q1 7 




No retponte 


0 


0 


0*3 




Veers Teacher Experience; 










0 ' ^ 


2*6 


1 


4 * £ 


1 
1 


1-3 


^' 13^2 


5 


4 « ^ 


1 


4-^6 


18*4 


7 


l£ * ^ 




7-10 


15*8 


6 


^^* u 




11 or okore 


50*0 


19 


50.0 


12 


No responee 


0 


0 




1 


HigheaC Degree Obtained: 










*^ ^ 


, 73.7 


28 


58.3- 


14 


HA 


18 ..4 


7 


37*5 


9 




0 


0 


0 


0 


No reaponee 


7.9 


3 


4.2 




Number of Previous NWREL Workshop Experiences; 










0' 


73.7 


28 


79*1 


19 


1 


23*7 


9 






2 


2*6 ' 


1 


4.2 


I 


3 


0 


0 


16.7 


it 


Reasons for Workshop Accendance: 










Satisfies a requirenenc or gives cr^ic 


, 63.2 


24 


54.2 


13 


Ochers were attending 


2.6 


1 


8.3 


2 


Superiors suggested I go 


5.3 


2 


8.3 


2 


Superiors gave me Che opportunity 


7*9 


3 


4.2 


1 


I was selected Co accend 


23.7 


9 


20.8 


5 


Accendance was paid for 


52.6 


20 


33.3 


8 


I really wanced Co learn 


73.7 


28 


50.0 


12 


I had a parCicular problem 


15*8^ 


6 


4,2 


1 


I heard . * * 


10.5 


■ 4 


25.0 . 


6 


Ocher 


23.7 


9 


29.2 


7 



er|c 
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The proportionate distributions of responses Cor both groups are 
essentially similar* There i^ere more females than males In each group* 
The average age of the RUPS participants was approximately the same as 
the average age of participants In the comparison group* The RUPS 
group had fewer years teaching experience as well as proportionately 
less people with master'^s degrees* Proportionately more people In RUPS^ 
had attended one or more NWREL workshops than In the comparison group, 
but more;people In the comparison group had attended two or more NWKEL 
works hops * The responses of the^two groups to the Item ^'Reason for 
Workshop Attendance" Indicated that the three reasons most often given 
by both groups for attending the NWREL workshops are; 
1* I really wanted to learn 

2* Satisfies a requlr^ent or gives credit ; 

3* Attendance was paid for 
In general, the participants* responses In both the ROPS and comparison 
group were similar for most of the variables under consideration* 

Procedures . ' 

Data Collection Procedures and Schedule; RUPS Treatment Group * 

Thft RUPS workshops were held on two consecutive weekends with two 

followup sessions six and twelve weeks later* The meetings were 

scheduled as follows; 

October 11 d;00 p*m* to 10;00 p*m* 

October 12, 13 8;^0 a*m* to 5;00 p*m* 

October 18 d;00 p.m* to 10:00 p*m* 

^ I . 

* * 

October 19, 20 8;30 a*m*.,to 5;00 p*m* 

-December 7 and 9;00 a*m* to 12;00 noon 

January 18 

30 
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The schedule Cor data collection was prepared prior to the 
workshop, based otv previous evaluation work conducted by NWREL* All 
the data collected from participants were from paper*and-*pencll 
.questionnaires* An evaluator was present at each of the workshopa 
and followup. sessions to observe and document deviations from prescribed 
training designs, the'degree of participant Involvement, overall uork^ 
shop climate and any apparent side effects. The data collection 
schedule Is presented la Table d» 

At-the first worlcstPtyp^tiKSffrtTrg, participants were assigned identl- 
fication ilumbers in order to insure confidentiality of responses* All 
data were identified and coded strictly on the basis of these ID 
numbersi At that meeting, prior co any training of thejpartlcipants, 
tVD questionnaires were administered»the Background' Questionnaire and 
the Problem Solving Orientation Questionnaire (PSOQ)* 

After participants completed the preliminary instruments; the 
workshop trainer introduced the materials » The prescribed training 
strategy as outlined in the trainer's manual was followed in the 
scheduling and presentation ot materials for the entire weekend* 
Participants worked alone or ih small groups of three to six* 

On October 14, at the end of the firsts weekend of training, 

participanta received a set of School Activities Questionnairea (SAQ) to 

If 

be administered to their classea during the following week (October 14 
tlitough 18)* Participanfis who had volunteered for EVPS training had 
been asked to arrange for a member of their school's professional or 
support staff to administer the questionnaire; teacherii were cautioned 
not to be in the room when the questionnaire was administered as it was 
felt their presence might influence responses. Confidentiality of 
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Table 6 



Data Collection Schedule 



InttruAent 



Background 
Qufttt lonnalrt 



Frobl'em 
Solving 
Orltntatlon 
Quttt lpnnalr« 

"XF50Q) 



School 

Actlvltlts 

Qutttlonnalrt 

(SAQ) 



Pott training 
Qufttt lonnalrt 



ProbloK 
Stattment 



Vftftkly Log 



Followup 
Quest lonaa ire 



Administration 
Dates 



avps . 

Prevorkshop 
Oct. U - 
pofftpcrtsoyi 
Preworkthop 
Sft(»t. 27 

JROFS 
Prfttfttt 
Oct* XX 
P osttet t 
0ct7"22 

Pretfttt 

Oct. 6 
Potttett 
Jan. 30 

ROPS 
Pretett 
Oct. X4-X8 

POtttfttt 

Jan. 20*24 

Pretfttt 

Sftpt. 27 
Potttfttt 

Nov. 29 

RUPS^ 
.Pottworkthop 
Oct. 20 



aup$ 

Pottworkshop 
Oct. 20 

RUPS 

PottworRshop 
and ninft 
weeks 

Oct. 25- 
Dec.^ 20 

RUPS 

Jan. Xd 



Data FiftXd 



Dttnographic and attitudinaX 
vaXuet 



Orientation of M^^^ici^antt 
towardt probXem soXving 



Adminis tared to the cXasset 
of participantt to mtature 
students^ perce^ciont of 
their teacher*t probXem 
soXving activities 



Assessment of Xearning 
experience, attitudes 
towards workshop and 
expected utiXization of 
skiXXs 

Statement of participants 
"backhome" RUPS project 



Frequency counts of probXen 
soXving activities and 
statements concerning the 
nature of the probXetn being 
worked on 



Assessment of Xearning ^ 
experience and participants* 
perceptions of positive and 
negative features o£ 
workshop 



Groups ColXected From 



RUPS GPS IffF ControX 



X , 
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responses vas maintained as students were not required to slgn^ their 
names and teachers vere requested not to iQok at the response sheets* 
All questionnaires and answer sheets vere then returned In a self- 
addressed, stamp^ envelope to NWREL where they were, coded by the 
teacher's Identification number* 

The second weekend of training started on schedule* The ""strategy 
and training instructions presented In the trainer's manual were 
followed with some minor chajiges In response to Individual and group 
needs and concerns* 

On Ckrtober 20, at the conclusion of the second weekend o^ training, 

participants completed the Posttralnlng Questionnaire and composed 

*, 

* 

, problem statements concerning their "backhome" RUPS project*^ These 
statements were collected by the evaluators at the workshops* ' Partlcl- 
pants also received copies of the Weekly Log forms and Instructions -for 
their completion* Weekly Logs were completed and returned to NWREL In 
enclosed self*addressed, stamped envelopes every Friday for nine weeks, 
October 25 through December 20* 



V 



The followup sessions were designed to reinforce use of sklTls 

and problem solving techniques learned during RUPS training* Partlcl* 

- ' \ ' 

pants reported progress made on the^lr back^ome RUPS proj^tts to their 

^ ^' ' ^ ' 

/ . 

training groups and together brains to rmed possible solutions* At both 

■ - 7' . _ . - 

follovup sessions the trajLners /followed the strategies and schedules 
prescribed by the trainer' s^^^nual with minor changes to allow for 
expression of Individual needs and concerns* Observers were present 
at the sessions to record deviations' from the prescribed training and 
any apparent side effects* No further data were gathered at the first 
followup session* 
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At the conclusion of the second followup workshop on January 18» 
the Followup Questionnaire was administered and the FSOQ was readmia* 
istered* Participants also received a second complete set of the SAQ 
to be administered to their classes during the following week» January 20 
through 24* The saute procedures were followed in the administration of 
the posttest SAQ as were followed in administering the pretest* 

Data Collection Procedures and Schedule; Comparison Group * The 
data collection schedule for the compariaon group is also presented in 
Table 6» page 32* The nontreatment control group met once on September 27 
for an orientation meeting* All participants completed Background 
Questionnaires on September 27* 

It was prior to the first meeting that the NWREL evaluation staff 
was notified of Tacoma's decision to withdraw from the iKJPS Impact atudy* 
Arrangements. were Inmiediately made for the control group to provide the 
'necessary compar'ative data* The PSOQ waa mailed to the comparison group 
during the week- of October 6* 

The GPS^ INF and nontreatment groups were involved in a study that 

participants' classes* The- climate inventory was composed of a seriea 
of subscales^ including the subscales of the SAQ used in the BOPS impact 
study; student responses to th^se particular subscales provided the 
comparative data for the RDPS study* Participants in the three groups 
received sets of climate inventories on September 27* It had been 
previously arranged that a member of the school's professional or support 
staff would administer the inventories* The inventories were returned 
in enclosed^ aelf-addressed envelopes by approximately October 11* This 
process was repeated on November 29 when complete sets of climate 



Inventories were mailed to all participants In this study* Admlnls^ 
tratlon procedures were the same and questionnaires were again returned 
In enclosed self^addressed envelopes by approximately December 20* 

On January 3, the PSOQ was mailed to the participant?. tn_Jhe j;ES^^ 
and liKP workshops as well as' members of the nontreatment group* 
Nlnety*flve percent of the questionnaires were returned by the end of 
January* ' ' . 
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CHAPTER THREE: RESULTS AND DISCUSSION 



The purpose of this chapter Is to report the data that were 
collected In this study and to discuss this evidence from the perspec*-/. 



tlve of the evaluation questions for the RUPS Instructional system. 
This chapter Is organized by evaluation question. The relevant data, 
analyses, and results are reported^ and discussed for each question, ^ 

Evaluation Questions Related to the Quality and Acceptability of the 
Content; Strategies, and Materials of the Instructional System 



Do RUPS trainees report aatisfaction with the training 
provided by the RUPS instruotionat system? 



The relevant data for this question come from the Posttralnlng 
Questionnaire administered at the conclusion of the workshop. These 
results are presented In Table -7. 

Slxty-flvft percent of the participants Indicated they felt thj£ 
vorlicshop was successful In presenting clearly understandable definitions 

*. ' \ ■ 

and descriptions of problem solving, and fifty-seven percent judged the 



workshop "to be successfuX'ln providing clear Information concerning the 
directions and rationales for the different sessions^ [.The majority of 
participants rated the workshop as successful In demanding original 
thinking (83 percent); offering new Insights, new ways df viewing old* 
problems (84 percent); and helping them understand the role and 
possibility of problem solving in schools (87 percent)* From results 
such as these It Is Inferted that the participants In the RUPS workshop 
vere^very satisfied with the content and strt^cture of^ the training, i 



:even ty- three percent of the participants summed up their 



experience In the BUPS vorkahop as positive; the same propoTtlon of 



I 



ERIC 
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Posttraining Questionnaire 
Satisfaction 



Quest iont 



How tuccessful do you feal tha workthop 



1. Providing claar information 
canceming directions and 
rationalfts'for tha diffftrant 
sessions? 

2. Offering new insights, q«w 
ways of viewing old probleme? 

3* Addreesing whet you thought 
were Important iseues/vital 
cone erne? 

Deoending original thinking 
on your part? 

5* Presenting clearly under- 
etandable definitions and 
descriptions 6f problem 
solving? 

6* Helping you understend the 
role end possibility of 
problem solving in echools? 

7* Maintaining your intereet 
throughout the vorkahop? 

24 » Meeting yoMt expectetions 
about vhet you i^rsonally 
wanted to get out of it? 



n 


I 


I 


3 


19 


.13 


X 


2.7 


2.7 


8.1 


51.4 


35.1 


n 




0 


13 


15 


5 


X 


10.8 


0 


35.1 


40.5 


13.5 



n 


1 


0 


5 


23 


7 


X 


2.7 


0 


13.9 


63.9 


19.4 


n 


1 


2 


10 


19 


5 


Z 


Z.7 


^.4 


27.0 


51.4 


13.5 



25. How cleerly did you under* n 
stand the workshop's overell Z 
objectives? 

26. How eucceseful do you feel n 
the workshop use in echiev* Z 
ing its overall objectives? 

29. Now thet the workshop is n 
over* how would you sum up % 
the experience? 

30. Would you reconBend this n 
workshop to a friend whose Z 
interests are like yours? 



Scale fUnge 



Neg 
1 



Neut 

3 



5 



1 

2.7 



1 14 
2.7 37.8 



18 3 
48.-6 8.1 



2 4 
5.4 10.8 



20 ir 
54.1 29.7 



n 


3 


2 


.9- 


20 


3 




8.1 


5.4 


24.3 


54.1 


8.1 


n 


Z 


1 


11 


16 


6 


Z 


5.5 


2.8 


30.6 


44.4 


16.7 



Neg" 

5 



Neut 

3 



Pod 
2 1 



2 

5.5 



2 

,5.4 



1 5 
2.8 13.9 



3 9 
8.3 25.0 



2 6 
5.4 16.2 



18 10 

50.0 27.8 



16 



8 



2 4 4 
5.4 10.3 10. 8 



44.4 22.2 



13 14 
35.1 37.8 



10 17 
27.0 45.9 



Totals 



X SP N 



3,57 ,80 37 

4.14 .89 l7 

3.46 1.10 37 

3.97 .?7 36 

3.68 .88 37 

4.08 .98 37 

3.49 1.02 37 

2.36 .99 36 



X SD N 



2.08 1.03 36 



2.19 .89 36 



2.05 1.13 37 



2.03 1.24 37 



^the scale range for items 25f '26> 29 and 30 is reversed; 5 indicated the most negative 
score while 1 indicates the most positive score. 
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partlcXpaats-lndlcatftd that- they would recoinmend\f?{/P5 to friends with 
similar Interests* Seventy-eight percent of the participants responded 
that they understood the workshop's objectives) but somewhat fewer^ 

s participants) slxty^seven percent) Indicated they felt the workshop was 
successful In meeting lt$ overall objectives* 

Participants' ratings of the workshop appear to decline slightly ' 
when participants are asked to respond on a personal level* Although 
participants tended to rate the workshop positively on Items from which 

^s^tis'factlon isMtiferred; responses tend to be somewhat less positive 

* 

when participants make judgments on the basis of personal criteria* 
Only 54 percent of the participants responded positively when asked to 

rate the success of the workshop In addressing what they thought were 

^ * 

Important Issues/vital concerns; 62^ percent rated positively the ■ 
workshop's success In maintaining their Interest* When asV;ed If they 
Celt the workshop was successful In meeting their personal expectations 
61 percent of the participants responded positively* 

These responses Suggest there may be a relative dissatisfaction 
concerning the personal relevance of the workshop) which in turn may 
have affected the participants*- rating of the workshop's success In 
maintaining their Interest and meeting personal expectations* The 
average response for items reflecting personal feellngS) particularly 
questions 3) 7 and 24)' are somewhat lower than the scores oC^the other 
questions* ThuS) assuming that satisfaction can be inferred from 
ratings of general succesS) the overall satisfaction ratings at the 
conclusion of the workshop are quite positive; all questions were 
rated positively by 60 percent of the participants or greater. But 
the results also suggest that if satisfaction were to be based solely 

39 
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on ratings of questions soliciting personal opinions » the respondents 
would still be generally satisfied but somewhat less enthusiastic* 

Do these i^evoegtionB persist over a S-month period? 

Some of the questions In the Followup Quest Ion nalre» which was 
administered three months after the end of the workshop » could be 
matched to questions In the Posttralnlng Questionnaire^ The responses 
to these matched questions were compared by means of uncorrelated t^tests* 
Table 8 contains the approprlaFe Followup'Questlonnalre data* T-values 
for all Items can be found In Appendix E* / 

The t*values for the tests performed on items 33» 34» 40 and 43 
were not significant* Thus» when questioned after a period of 3 months^ 
i?£/P£^ participants tended to maintain their perception^ of the workshop*s 
success'ln addressing important Issues and vital concerns^ presenting a 
clearly understandable description of problem solving^ and achieving 
Its overall objectives* There was no change In the percentage of 
participants who responded that they would recommend the^ RUPS workshop 
to friends and colleagues* The results Indicated that after three 
months there was no significant change in the satisfaction ratings of 
RUPS participants on these variables* 

The t-values for questions 32 (tj2 " -2*06, p < *05) and 42^ 
(tj2 ' 2.962, p < ^05) Indicated that there was, respectively, a negative 
change in the participants' perceptions of the workshop*s success in 
offering new Insights and ways of viewing old problems, and significantly 
fewer people summed up their workshop escperience positively* 

^The 'rationale for using uncorrelated t-tests in similar situations is 
discussed in McNemar; Psychological Statistics^ 1962, p,* 85. 
^The scale range for Item 42 is r.eversed* 
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Table 8 



Followup Questionnaire 

Satisfaction 



Questions 


Scale EUnge 


Totals 


Keg tteut Pos 
1 2 3 4 5 


X SD K 


How auccessful do you feci che^workshop 

32. Offering new Inalgbts^ now vays n 
of viewing old problama? (2)* X 

33. Addressing laporcenc iiau«5» n 
vicel concerns? (3) X 

3^* Presenting a cltarly under-* n 
scendable d«acripcion of % 
problem solving? (5) 


1 3 12 12 9 
2.7 8.1 32.4 32*4 24*3 

2 5 13 16 1 
5*4 13.5 35*1 43*2 2.7 

3 2 11 15 6 
8.1 5*4 29*7 40*5 16*2 


3.68 1.03 37 

3V24 j93 37 

/ 

3*51 Jl.lO 37 


39. Compared to other workshops ' n 
you have act«ndad» how X 
successful do you f««I chii 
workshop w«s Uk meeting your 
expectations about what you 
personally wanted to g«t 

out of it? 

40. How succeseful do you feel n 
the workshop was in achieving % 
its Overall objectives? (26) 

42. Now that the workebop is n 


Keg^ Neut PoS 
5 4 3 2 1 


X SD K 


3 6 3 18 ' 2 
8*1 16.2 21*6 48*6 5.4 

1 4 10 21 1 
2.7 10.8 27.0 56.8 2*7 

1 15 16 12 -2 


2.73 1.07 37 

2.54 *84 37 
2.75 *87 36 


over, how would you sum up % — 
the experience? (29) 

43+ Would you rtconunend this n 
workshop to any of your ' % 
fri«nds or colleagues? (30) 

44. In relation to your expendi*^ n 
ture cf time and/or noney^. X 
how valuable has the work-* 
shop ba«n to you? 


2 5 5 U 11 
5*4' 13,5 13.5 37.8 29*7 

i 4 9 16 7 ■ 
2*7 10*8 24.3 43.2 18*9 


2*27 1.19 37 
2.35 2*01 37 



^The ttumb«ra in par«nthesas indicate tsatchins items from th« Fosttraining Questionnaire. 
^The seal* rang* for items 39» 40, 42-44 ^is reverted; 1 indicates the most positive 
score while 5 indicates the most negative score* 
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These results indicate that^'^he tendency for participants to respond 
less positively when questioned on a personal level is continued* The 
workshop participants have, presumably, been using RUPS for three months* 
Based on their exi^eriences they have developed a more realistic under- 
standing of RUPSj its strengths, and its limitations* 9ut whereas 
participants tend to i^espond less positively when Judging the worlcshop 
on personal criteria, their attitudes about the training and quality of ^ 
the materials have not changed* 

Do trainees perceive' the training in RUPS as being 
, useful and applicable to their work? 

The information source for this question is the Fosttrainihg 
Questionnaire* The results are presented in Table 9* ' The majority of 
the participants, 84 percent, tesponded that they felt the workshop was 
successful in helping them understand the^'role and possibilities of 
problem solving in schools, and 81 percent rated the skills andr concepts 
of, RUPS AS useful* Fifty-seven percent responded that they planned to 
4i0e^RKP5-a&-^n— integtai— part— of— thair-wo^ results it is 

inferred that, immediately after the workshop, participants tended to 
perceive training in RUPS ^s both useful and applicable to their work*^ 

This result vas further illuminated , by questions regarding the 
applicability of RUPS to specific groups* A large proportion of parti- 
cipants, ranging from 57 percent to 79 percent, saw RUPS as applicable 
and useful in work with students, teachers and others, but only half, 
49 percent, of the participants saw RUPS as useful and apptic^jble to 
work with superiors* This would suggest that whereas, the trainees could 
visualize the role and usefulness of RUPS training, skills and concepts 
in dealing with students, teachers, and others, they were less sure o£ 

42 ' ' ' 
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Table 9 



Posttralnlng Questionnaire 
Utilization and Application 



Questions 


Scale Range 


Totals 


Heg Heut Pos ■ 
12 3 4 3 


K SD H 


How 9ucce»ful do you f««l the workshop 
WAS In': 

6* Helping you understand the , n 
role end possibility of % 
problcQ solving in.schoolsT 

^ 8. Providing useful skills end n 
concepts for working with X 
oth^s outside your profes* 
sional life? 

9* Providing infonsation with n 
precticel applicetion for X 
your work with .students? 

10* Providing infotmetion with . n 
prftcticel applicetion for X 
your work. with teachersT 

11. Providing information with n 
practical application for X 
your vork with superiors? 


0 2 4 20 11 
JO 5*4 10*8 54.1 29.7 

0.3 8 17 8 
0 8.3 22.2 47*2 22.2 


4*08 .80 37 

3*83 .88 36 

c 


1 1 6 20 9 
2.7 2*7 16*2 54*1 24*3 

2 0 14 ZO 1 
5*4 0 37.8 54*1 2.7 

1 4 14 17 1 
2.7 10.8 37.8 45*9 2*7 


3.95 *88 ^37 
3.49 *fiO 37 
3*35 *82 37 


.27* How useful do you fe«l the n 
skills and concepts you have X 
learned are in problem 
solving? 

28. In all honesty, how much do n 
^ou plan to use the ideas, % 
skills> and/or materials 
presented in this workshop 
as an integral part of your ■ 
work? 


^ Heg^ Heut Pos 
5 4 3 2 1 


X SP H 


0 2 5 14 15 

0 5.5 13.9 3S.9 41*7 

1 2 13 21 0 
2.7 5*4 35.1 56*8 0 


1.83 *88 36 
2*54 *73 37 



^he scsl^e range for items 27 and 28 is reversed; 1 indicetes tW« most positive score 
while 5 indicetes the most negative score. 



Its use and applicability (although the responses were still In a 
positive direction) to their work with superiors* However^ this. result 
may be due to the fact that the RUPS materials are geared for classroom 
use* Thus» It would seem reasonable that participants found It more 
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useful in that setting* Since the relationship of RUPS to work with 
superiors Is less clear, It Is quite possible that fewer people would 
see Its applicability* 

Do these perceptions persist over a S-montk p^iod? 

The data sources are the matching Items from the Fosttralnlng 
Questionnaire and the Followup QuestlQnnalre* The results are presented 
In Table 10* Uncorrelated t<^tests were performed on all Items In Table 8» 
page 41* T*values are presented In Appendix Item 36 (tyj • *2*10> 
p < *0S) was the only Item found to have changed significantly » Indicating 
that there was a slgnlflcsnt decrease of perceived success of the workshop 
In providing Information with practical applications for work with student 

Table 10 



Followup Questionnaire 
Utilization and Aiifpllcatlon 









Scale lUnge 


Totals 


Qu«ttioiit 




1 


2 


Neut 
3 


4 


90S 

5 


X 


SD 


N 


How 
wit 


tuccesiful do you f««i the workshop 

In: 


















35. 


Providing uaeful skills «nd 
concepts for working with 
othtrs outsid« your profes-^ 
siottil liUI (8)* 


n 
X 


2 

5.4 


6 

16*2 


9 

24.3 


14 

37.8 


6 

16.2 


3.43 


1 12 


37 


36. 


PtovidinS infomtion with 
prActicAl Application for 
your work with students? (9) 


n 
X 


1 

2.7 


7 

18.9 


10 

27.0 


12 

32.4 


7 

18.9 


3.46 


1.10 


37 


37. 


Providing infor»tion with 
prActicAl Application for 
your work with tttchsrs? (10) 


n 
X 


2 

5.4 


4 

10.8 


8 

21.6 


17 

45*9 


6 

16.2 


3.57 


1.07 


37 


38. 


Providing infor»tlon with 
practical application for 
you^ work with superiors? 
(11) 


n 
X 


2 

5.4 


5 

13. ^ 


16 

43.2 


12 

32.4 


2 

5.4 


3*19 


.94 


37 



*Thtt nuabers in parenthetes indicate matching Items front the ?osttralnlng Questionnaire. 
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In addition there was an open-tended question on the Followup Question-* 
nalre which asked, ^^Now that you have been away from the workshop for 
three months, have you changed your opinion about how useful It was?^^ 
A complete list of participant responses Is provided In Appendix 

These results Indicate that RVi^S trainees, when questioned three 
months after the workshop, did not change their positive perceptions 
of the usefulness and applicability of BVFS training In relation Co 

work «wlth teachers, superiors and others* The tendency for participants 

■■ ■ ' « 

to be less sure (In comparison with their responses relating to teachers 
and others) of the usefulness and possible applications of RVPS to work 
with their superiors was continued. However, the participants^ ratings 
of the usefulness and , practicality of RUFS for woi^k with students 
decreased although the responses remained positive* This change, 
however. Is not totally unexpected'. In the three months since the 
workshop, trainees have had the opportunity to use the BJPS process and^ 
experiment with the skills and concepts In their classroom* The 
participants' responses may reflect a more realistic understanding of, 
WPS process based on their classroom experiences with It. Participants 
have apparently^learned to adjust their expectations of the practical 
application and usefulness of RVPS to the dally demands and limitations 
of their work situations. 

When questioned directly about their opinions of the usefulness of 
the workshop, 88 percent of the participants responded that their 
opinions hiad not changed since the workshop. Their elaborations of 
their responses to this open-ended question generally Indicated 
positive attitudes toward the RVPS process and Its components. This 
measure of self report generally supports the results of the Followup 
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Questionnaire Items; opinions of the usefulness of RUVS generally did 
not change^ although participant dplnlon about the usefulness of BUPS 
for work With students did change* 

aTtat are the significant eida effects that result from 
the use of the system? 
* 

The data sources for this question are the observations of the 
evaluators /observers who were present at each workshop site and the 
results of Form 2 of the Weekly Lo%*\ 

A side effect of RUPS training that was observed and documented at 
both the conclusion of the workshop and at the two fpllowup sessions was 
the high degree of participant enthusiasm foic the training* The 
observers attributed this enthusiasm to two causes; (a) the success of 
the training In providing a novel perspective aixl alternative way(s) of 
approaching problem solving, and (b) the reln£or&lng Interpersonal 
dynamics of the training/learning grtftips* 

Another possible side effect of the training was observed In 
participant responses to Form 2 of the Weekly Log* These results >• 
Indicated that i?£/P5 .participants tended to select relatively complex 

problems for their Improvement projects that required commitments of at 

* 

least one month's time for their completion* The majority of people 
(fo percent) reported working on the s^me Improvement project for at 

least one month* However » as there yere no comparative data for these 

* * 

results^ .It Is not possible to determine whether thls^ effect Is Indeed 
due to RUPS training and not the reactive effect of this data gathering 
Instrument. 
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Does RUPS compare advantegousty to competitive ways 
' of meeting the same outcome obgeotives? 

A limited Educational Resources Information Center (ERIC) search 
was conducted for the purpose of Identifying problem.solvlng training 
packages or Instructional system that offered alternatives to WPS* 
Using th&. descriptors^ "problem solving^ action research^ force field 
analysis^ vorkshop?^^ taacher workshops^ systei&s anlaysls^ systems 
approach^ and systems concepts/* the search Identified 74 references* 

This search Indicated that many Individuals and consulting 
organisations have Included i?£/P£^*l Ike problem solving techniques as 
resources In their work* It provided evidence of the development of 
a variety of materials for single^ short-term activities (half^days or 
less) based on force field analysis^ action' research^ or systfems 
approach to problem solving* Tti^enty Exercises for the Classroom 
developed by the NTL Learning Resources Corporation (Mlll» 1972) Is an 
example of this type of effort* In this senae» many of the. training 
packages Identl'fled In the search are comparable to RUPS In that at 
least some of their activities and problem solving activities are like 
^those used' In RUPS* The concepts and activities surrounding action 
research and force field analysis are by no means unique to RUPS* 

In terms of complete and comprehensive prepackaged training 
programs that emphasize both problem solving and Interpersonal and 
group dynamics (as does RUP^)^ We were unable to Identify any alterna*^ 
tlves to the RUPS Instructional system* While a variety of commercially 
available systems and activities exist that have some objectives In 
ccMsmon with RUPS^ ^our search did not provide evidence for any compre* 
henslve and comparable systems that could provide competitive ways of 

J 

meeting all the training objectives of the HUPS Instructional system. 



However) this does not Imply that there are no critical competitors 
which accomplish the same outcomes as RUPS; It only implies that these 
materials are not available as training packages* 



Questions Related to Lgng^RanRe Training Effects 

To evaluate training in terms of outcomes or effects acKleved Is a * 

i 

complicated undertaking* One of the problems which complicates the task 

Is disagreement over the type of training that Is most useful or has the 

greatest Impact* This disagreement may be due largely to differences In 

the values applied to the training systaos* Is It more useful to 

develop efficient forms of training that are j>t limited applicability 

or broadband forms of training that may be applied In a variety of 

situations? In the opinion of the evaluators and ss supported by^ 

external evaluation revlewss RVPS falld In. the second of these two 

* 

categories* 

To evaluate the effects of RUPSj thens'ls similar to evaluating 
grsduste courses In such areas as research methodology) techniques of 
counsallng) and apptoaches to management* When the values to be applied 
deal with Impact on clients > whether they be students, patients > or 
subordinates) the primary evaluation problem Is to choose the outcomes 
on the basis of an explicit rationale* 

There are at least three options for choosing evaluative criteria* 
The first Is on the basis of specific objectives dealing with client 
behavior* While thl« would clearly be appropriate In the efficiency 
oriented training programs broadband training programs frequently do 
not have such objectives* 

A second, basis for choosing impact criteria Is by authoritative 
decree* For examples standardized achlev^ent test data may be used 
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to evaluate teacher training programs because someone In a position 
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power has demanded that they be used* If the logical links between 

t 

the training and these crlterl££*are nonexlstant, a costly ^plrlcal 
Tstudy may not be warranted. " - 
A third alternative Is to select outcomes 'based on a model pre- 
dicting the expected impact on students. This technique was chosen lii 
' the RUPS Impact evaluation. Since selectJns criteria ^tnd developing 
a design based upon a model Involve a series of Inferences, an ^ 
explanation of these' Inferences Is provided through the vehicle- of 
path analysis. ?ath analysis was chosen as an analytic model because 
dt makes explicit the theoretical assumptions underlying the evaluation 
design. 

The firs't step in applying path analysis is to create a graphic 
display of the hypothesized causal relations operating aniong a set. of 
variables. A general path display for evaluating RUPS is presented in 
Figure 1. The lines connecting sets of variables represent hypothesized 
causal relationship^ between the connected variables, while the arrows 
indicate the direction of the assumed causation. Figure 1 makes clear 
the assumption that the final outcome measure in the sequence is directly 
or indirectly determined by all of the preceding variables, not Just 
training. This is particularly important, since randomization was not 
achieved in the present investigation. Figure 1 is a general display 
from which a series of path diagrams may be derived. A path diagram 
contains specific variable levels whereas a general path display may 

> 

iarlude. categories of variables as terms. Categories of variables ' 
included in the general path display are teacher attitude, teacher 
behavior, and student attitudes. 
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Figure General Path Model for Che RUPS Impact Study 



The evaluation questions which are addressed In the remainder of, 
this report can be placed within the context of this general^ path mode^* 
It should be noted that formal path analysis proced^ures ifere not appllel^ 
In addressing all of the evaluation questions which follow^ but th^ 
model Is Intended to reflect the logic In the order In which the 
questions are addressed* 

Attltudlnal Effects * The data sources for the following evaluation 
question correspond to sections of the Posttralnlng and Followup 
Questionnaires that assessed participant perceptions of their "working 
understanding^* of workshop skills and concepts* 

Do trainees * perceptions of their working understanding 
of the materials change over a S-^month period? 

> Table 11 presents these data and a statistical analysis by means 
of correlated t*tests* 

The mean score on all items on the Fosttralnlngr Questionnaire was 
greater than three» suggesting that a majority of participants felt they 
understood the materials but did not feel that they could apply the 
skills iMnedlately after training.^ These data indicated a general 
confidence on the part" of the RUPS participants with regard to their 
working understanding of the skills and concepts of RUPS trainings but 
participants appear to be less confident of their ability to apply 
the skills* 

Similar results were obtained three months later when the items 
were readmlnlstered on the Followup Questionnaire. Only the item 
assessing the participants* working understanding of "spotting and 
analyzing major results of data cQllecCed" was found to have changed 
significantly. This change was In a positive direction Indicating that 
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Table 11 



Knowledge Items; 
Fosttralnlng versus' Follouup Questionnaires Correlated t-Tests 





Foe c era J 


ning 


Followup 


Difference 








X 


SD 


X 


SD 


X 


SD 


c-yalu« 


df 


How utll do you f««l you hAV« a 
working uiid«r#candinK of: 


















Applying ch< 4'guld«Iin< 
criceria for vricing a 
problem scactmeac 


3.69 


.58 ■ 


3.63 


.84 


-.06 


,68 


-,49. 


34 


Using ch« force field 
diagnostic technique 


3,92 


,28 


•J ,OJ 


. 






-1.00 


34 


^ Selecting end creecing 
inecrusence for deca 

gachering 

* 


3,63 


,55 


3.60 


.65 


-.03 


.86 


*,20 


34 


Using criteria for deriving 
laplicecions fron research 
findings 


3,24 


.96 


3,26 


.79 


.02 


1,14 . 


.15 


33 


Spoccing and analyzing major 
resuXcs of daCa collecced 


3*33 








, jl 


.01 




32" 


Breinscoming accion 
alcernacives Co meeC 
iaplicecions derived 
fron findings 


3,68 


.68 












33 


Applying guidelines for 
plenning end lapIeQenCing 
accion alcernecives 


.312 


.96 ' 

'J 






ns 


1.21 


/ 

/ .28 


34 


£\reIuacion solucion plans 


3. 12 


.91 


3.47 


.66 


:35 


1.07 ■ 


1.92 


33 


Paraphrasing incetpersonal 
comunica c ions 


3.71 


,67 


3,71 


.57 


^ 0 


.50 


0 


3r4- 


Using concepcs of giving 
end receiving feedback 


3.68 


.73 


3.79 


:54 


,11 


.81 

r.oo 

/ 

1,26 


.85 


33 


Diagnosing ceanwork relacions 


3.56 


.75 


3.59 


.66 


.03 


.17 


33 


idencifying end evaluaclng 
small group dynsaiCs 


3.29 


,94 


3.4^4 


.79 


.15 


.68 


33 



^the following scele was used: 

1*1 don^c understand Che concepc end could , no c apply che skills 

2 « I don*c understand che concepc buc~could apply che skills 

3 « I underscand che concepc buc could noc apply che skills 

4 « I.underscand che concepc and could apply che skills 
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the participants* percep^tions of their ability to apply this skill had 
increased* This effect' may be due to participants* use of the data 

gathering inj^ument^^ 

practice and classroom experience RUPS trainees had had since the 
workshop* As a result of their use- of both the instruments and the 
RUPS process > participants^ may have had more confidence in their 
performance of WPS skills and activities* Thus» the data indicate 
that trainee perceptions of their working understanding of the workshop 
materials and concept were very positive when assessed immediately after 
the workshop^ an^ generally did not change over a 3^onth period* 

Do RUPS trainees change in theiv orientation toward 
pvobtem solving? 

It was hypothesized that as a result of trainings participants in 
RUPS would report an increased probability of engaging in problem solving 
behavior in a^variety of situations* This hypothesis was tested using 
the four PSOQ scales administered prior to and three months following 
the workstiop* Correlated t-tests were performed on the data* The 
results are presented in Table 12. 



Table 12 

Correlated t^Values 
PS0Q~i?I/P5 Treatment Group 





Pre 

X sb 


Post 
X SD 


Difference 
X SD 


t-*Valua 


df 


p-» Value 


Classroon 
Situations 


36.35 


6*18 


40.11 


4*85 


3.76 


5.35 


3.94 


36 


.01 


Noac lass room 
Situations 


47.49 


5,22 


47.30 


4.94 


-.19 


5.17 


-.22 


36 


.39 


Task 

Accomplishment 


35.24 


6* 50 


37,97 


5.65 


2.73 


5/99 


2.77 


36 


.01 


Interpersonal 
Relationships 


49*54 


5.01 


50.30 


5-20 


.76 


5.02 


-.92 


36 


.26 , 



The data Indicate a statistically significant Increase In the self- 
reported probabilities of engaging In problem solving activities for two 
of .th% four subscales; classroom situations and task accomplishment* 
The training was apparently Instrumental In changing participants^ 
dispositions too, or their perceptions of their dispositions to engage 
In probelm solving behaviors In the classroom and specific tasks* Both 
presumably impact directly on a participant's problem solving attitude 
to^rd, as well as behavior In, schools and classes* However, It Is 
possible that the same variable' Is being measured by both scales as , 
these two scales contain many of the same Items* A variable from th€^ 
combined scales may be referred to as participant orientation toward 
task related problem solving In the classroom* The t*values for the 
other ^wo scales were not significant* RUPS training apparently did 
not affect partlclpanrt perceptions of their interpersonal problem solving 
orientations or orientations to problems other than classroom related 
problems* While these differential results may be explained by the 
strong classroom orientation of the RUPS materials, there Is an equally 
compelling psychometric reason for such a finding* Reference to the 
technical appendix on the psychometric characteristics tAppendlx C) will 
clearly show that the reliabilities of the two scales that did not show 
' change wera much lower than for the scales that did show change* 

Do RUPS traimes beome move able to identify problem 
that need attention at^d more willing to attend to problem 
solving in a systematio way than do teachers who have not 
received training? 

It was hjrpotheslaed that the posttralnlng orientation toward problem 
solving of participants In the RUPS workshop would be greater than the 
posttralnlng orientation among members of the nontreatment control groups 
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This question was addressed through the technique of path analysis. 
The path diagram reflecting the model employed is presented in Figure 2. 
Figure 2 is -derived from the model presented in Figure 1» page 50* 



Agei 



Sex 



Class Size 



RUPS 

Training^ 




Teacher Orientation 
Toward Problem Solving: 
Task Accomplishment^ 



Figure 2* Path diagram for evaluating the effect of 
RUPS training on teacher orientation toward problem 
solving* 



Variables were selected for this path diagram on the basis of a number 
of considerations* The most Important variable is the teacher attitude 
measure c^onceming task accomplishment » which was ct^osen for the path 
analysis because of its relatively high reliability in comparison to 
other PSOQ scores* The RUPS training variable (dummy variable where 
RUPS * 1 and control » 0) is included ^ the measure of treatment* 
Additional variables thought to influence teacher attitude were age» 
sex» and ^\ass size* It is assumed that these four independent 
variables were all measured very reliably^ and» thu$» were appropriately 
, included in the path diagram* ^ 

The regression equation corresponding to this path diagram is as 
follows': * 

By solving this linear regression equation It is possible to make 
estimates -of the direct-effects (path coefficients) of prior variables 
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upon subsequent variables* While these path coefficients do not In 
themselves Indicate causality they can be used to assess whether or not 
the data are consistent with a prior ^au^l model* Table 13 presents 
the standardized partial regresalon coefficients or path coefficients 
for each of the variables In Figure 2 and the multiple correlation 
obtained for explaining posttralnlng teacher orientation toward problem 
solving; task accompllshmeht* ^ , 

Table 13 

Multiple Correlation and Standardized Partial Regresalon 
Coefficient's for Explaining Posttralnlng Teacher Orientation 
Toward Problem Solving; Task Orientation^ 



Exogenous 
Variable 


Multiple 
Correlation 


Standardized 
Partial Regression 
Coefficient . 


p-Value 


Age 


*4d 




.01 


Sex 




-*23 


.19 


Class Size 




.21 


.23 


Training 






.37 



^Only subjects for whom thete were complete data were uaed In thla 
analysis* (N*26, RUPSi N"10, Control; N-36, Total) 



It Is apparent through examining Table 13 that RUPS training did 
not have any direct effect on teacher orientation toward problem solving 
as the corresponding coefficient was -*ld* The multiple correlation ' 
resulting from regreaslng posttralnlng teacher orientation toward 
problem solving; taak accomplishment on the four variables In the path 
model was *4d* The-otily significant path coefficient was for age 
(p < .01), Indicating Chat younger teachers saw themselves as having 
stronger orientations toward problem solving. 
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Behavioral Effects > The Information from Form 1 of the Weekly Log 
was used to answer the following two*part question* 

Do RUPS traineea di9cu8$ their pvobtm aotving knowledge 
and akitta in theiv local schools? 

Do RUPS trainees uge their problem solving h%ou)ledge and 
skills in theiv local schools? 

The mean frequencies and standard deviations of reported activities 
with student;? £i^e presented In Table 14, and those with faculty and 
colleagues are presented In Table 15. ^ 

The rej^ults Indicate that RUPS partlcpahts reported engaging In 
a mean frequency of 18.13 RUPS-like. problem solving activities per week 
with their students* The mean frequency of activities per week with 
faculty and colleagues was only slightly higher (X * 19.36), suggesting 
that RUFS participants perceived that they used WPS'^liit^ skills about 
as often with their peers as with their students. This evidence 
Indicates ''that the participants In the RUPS workshop do use problem 
solving skills In their work In the classrooms and with faculty and ' 
colleagues. However, since there were no comparative data that can be 
used to Interpret these results either from other training groups or 
from the BUPS participants prior to training. It cannot be determined 
whether these results reflect changes In problem solving beha\^lor that 
are due to training, or represent typical teacher performance. 

^ It Is Interesting to observe the trend that Is apparent In the 
weekly average frequency of activities. The period during which RUPS 
skills were most frequently reported used occurred In weeks 4, 5 and 6. 
This pjattem> however^ may be an artifact of followup sessions and the 
possible reactive nature of the data collection procedures. The first 
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T*bl« U 

Uevkly Los — Focua of Activity: Studenta 



A4:tlvlty 


Week 1 


Weak 2 


Ueek 3 


Uc«k 4 


threk 5 


Ue«k 6 


Wtek 7 


U«t«k 8 


Ueek 9 




X SO. 










. S SD 




X SD 


X SD 


EX 


TrloJ Co Kidn an nctual ■tate*' 

■kCnt of Wlk3t 4 pATt Iciil 4r 

pcoblw wac* 


1.1* 3.32 


1.68 2.00 


2.44 6717 


4.35 11.30 


3.65 11.47 


3.32 9*94 


1.44 2.76 


1.79 4.27 


1.53 3.82 


21.34 


Tried to Identify jkOtentlal 
dlfflcultle* and *ldB 

t C l- U 1 • ^XMf t It 111* F 

pcoblen. 


.ft5 2. IB 


.97 2.41 


1.^ 2.35 


.85 2.47 


1.38 4.14 


l.SO 5.74) 


.32 1.01 


1.35 2.85 


.79 2.06 


9.24 


Hade or got *wc t«*ta or 

1 lonnsl re to col lec t 
dat4 about ^ problem. 


1.33 3.50 


2.09 3.74 


2.89 6.20 


4.59 11.24 


4.03 11.42 


3.32 9.93 


1.47 2.72 


2.00 4.29 


1.85 3.85 


23.56 


CollfiOted and 4nalyz«d dat* 

results *pplied to a ^ 
P^trtlcular problua. 


.58 1.41 


.62 1.01 


1.53 3.95 


.85 2.50 


1.47 4.16 


1.41 5.67 


.32 .91 


1.29 2.77 


.85 2.18 


8.92 


'niought of or lljitvd different 
«olutlontf axid plans of action 
to ftolvc % problcM. 


.74 2.23 


1.29 A. 50 


.97 1.38 


1.29 5.64 


1.12 3.7<^ 


l.SO 5.66 


.59 1.96 


.62 2.01 


.97 3.97 


9.09 


Tried out possible solutions 
and plains of ac c Ion to solve 
a problem* 


.1ft .63 


.SO 1.74 


.74 


.44 1.99 


.47 1.16 


.76 2.62 


.23 1.05 


.35 1.39 


.56 2.00 


3.72 


Evaluated or revised solution 
Plan* for solving 4 |>roble«. 


.6a 3.07 


.47 .75 


1.56 3.64 


2.10 6.66 


1.53 4.11 


2.3* 9 . 51 


1.00 2.64 


.65 2.13 


.50 1.93 


10.87 


Discussed general problem 
sol V 1 ng^st r^ t eg 1 e» . 


.35 1.15 


.41 .89 


.41 1.13 


.65 2.09 


.18 .39 


I. 00 3.81 


.21 .77 


.68 2.06 




4 .18 


Otecus<«d specific Problem 

£0 1 U 1 nff DlVk Cfcclu 1* _ 


1.00 2.16 


1.38 1.97 


3.00 9.00 


4.12 10.56 


2.71 4.54 


3.76 10.01 


1.44 2.71 


1.59 3.95 


1.76 3.85 


20.76 


Suggested general problem 
solving strategle*. 


.47 1.05 


.53^05-* 


1.35 3.73 


£.03 2.58 


.62 1.30 


1.06 3.80 


.41 1.26 


.79 2.11 


.79 2.17 


6.75 


H«commendcd specific problem 
solvlthg ptoce^urea. 


^.85 2.12 


1.47 2.43 


2.77 9.01 


2.29 9.92 


3.29 11.26 


3.50 9.89 


K39 2.67 


1.32 3.89 


1.56 3.77 


18.44 


Attended to pi:'Dbl«is of feed- 
back and Interpersonal 
comunicat Ions. 


.32 1.15 


.59 1.84 


1.32 3.80 


.76 2.41 


1.06 3.dB 


1.03 3.86 


.19 .97 


.73 2,12 


.82 2.21 


6.92 


Actended to problems of team- 
work and group Interactions. 


.77 2.15 


1.18 2.22 


1.06 2.26 


1.82 4.15 


2.94 9.27 


2.71 6.88 


.71 1.99 


1.18 3.98 


1.41 3.77 


13.78 


Thought about or engaged In 
specific prob^en solving 
activities. 


.38 1.39 


.21 .54 


.91 2.22 


.38 1-10 


.82 2.38 


1.03 4.09 


.48 .52 


.68 2.17 


.71 2.01 


5.30 


EX 


9.67 


13.39 


21.67 


25.52 


25.27 


28.28 


10.00 


15.02 


14,39 
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Tibia IS 

Ucttkly Lo|»Fi>cus of Activlryt ' Faculty Aod Collednucs 



C5 



U1 



Activity 


liaek 1 


tfeek 2 




Itosk 4 


Week 5 


W«ek 6 


We«k 7 


tieek a 


Ueek 9 




X SD 


X SD 


X * St> 


X SD 


X ^ 


X SD 


X SD 


X SD 


X Sl» 


Trl«d to «akc an actual atat«> 
^nt of whdt ■ p«ttlculat 
ptobica vaa* 




.•• 1.30 


..91 1.54 


3.41 10.22 


3.26 11.42 


3.15 9.63 


1.23 2.77 


.76 2.13 


1.12 2.24 


15.66 


Ttt^ to identify potential 
dlfflcultlca and «lda 
conctttnliif a p^tticulat 
ptobleti. 


.41 .99 


.65 1.07 


.71 1.29 


1.40 3.41 


1.41 4.0s 


1.59 5.77 


.50 1.13 


1.21 2.76 


.6a 1.90 


8.56 


Mdde ot %ot aoK tcata ot 
^titi«tlo>inaitca to collect 
data About ■ ptoblftia^ 


.71 2.1 J 


.65 1.2a 


.•2 1.40 


2.21 9.3< 


l.a2 7.35 


2.56 9.54 


1.29 3.06 


.73 2.27 


.76 2.03 


11.55 


Collected and aaalyxed data 
Afwil thou^t ikbout Ihm tha 
tcsults applied to a 
patticulat probi««« 


.41 .96 


.76 1.16 


.60 1.65 


1.44 *.66 


.97 2.17 


1 .26 4. 14 


.56 1.5a 


i.ia 2.a6 


.76 2.i5 


8.02 


Thought of ot Hated dlffetent 
uolutioiif attd pinna of action 
to 0olve a problca. 


.79 1,99 


.65 i.ii 


.•2 1.10 


2.50 1.50 


2.15 11.29 


2.47 9.4a 


1.12 2.76 


.>4 3.98 


1.21'^. 80 


13. 35 


Tticd out posalble solution* 
an<J plan* of action to solve 
a ptoblca^ 


.» l.OS 


.29 .61 


.62 1.41 


.73 2.33 


1.29 4.13 


1.06 4.01 


.41 .89 


l.ZL 2.75 


.73 2.03 


6.72 


Evaluated ot taviaed aolu^lon 
plana fot solving ■ ptoble«.» 


.62 2.00 


.56 .96 


«59 KOi 


2.50 a. 52 


.59 .16 


2.47 9.44 


1.03 3.13 


.91 3.96 


.73 ' 2;ia 


10.00 


Diacuaaed general ptoblev 
solving attategiea* 


.41 1.02 


.21 ^55 


.62 1.11 


.73 2.31 


.44 1.46 


1.32 5.73 


.59 2.10 


.71 2.10 


.38 .82 


5.43 


Diacussed specific Ptobl«tt 
solvlag pn>cediit«0. 


.41 2.6« 


2.09 3.63 


2.62 6.50 


3.00 6.00 


3.53 7.05 


3.35 9.9t 


1.65 3.12 


l.a2 4.42 


1.59 2.63 


20.06 


Suggeat«d gunetal ptobl«ai 
living attategiea. 


.59 l.» 


^1.29 2.70 


2.21 4.46 


l.aO 4.50 


1.71 3.92 


1.6a 5.79 


.9X 2.35 


l.M 2.70 


l.ia 2.37 


12.75 


ftecoaaended specific ptoblev 
solving ptocedut«a. ^ 


1,00^2.19 


1.23 1.61 


2.feO 5.94 


3.56 9.95 


2.76 6.33 . 


3.73 10.25 


1.35 2.94 


1.65 4.13 


1.26 2.4a 


19.34 


^Attend^ to ptobltfn of fued* 
back and Intetpet^nal 
coH^uii i c A t ions * 


.50 1.46 


.76 1.65 


1.6a 3.04 


1.56 3.31 


1.56 3.69 


2.ia 6.27 


.50 l.Oi 


1.15' 2.43 


1.21 2.40 


11.10 


At^ttidad to ptoblcne of tear 
uotk gtouf^ intetactiona^ 


1.27 2.06 


1.59 2.21 


2.12 3.07 


2.15 9.M 


3. 50^ a. 26 


4.06 10.35 


1.00 2.72 


1-S6 4.34 


1.85 4.35 


19.80 



Ttiought about ot engaged In 
specific ptot>le« solving 
activities. 


.77 2.44 


1.06 2.27 


1.06 2.52 


1.59 3.43 


1.6a 4.41 


2.26 6.70 


.65 i.a6 : 


1.44 3.0a 




1.38 3.25 


11.89 




9.21 


12.69 


ia.26 


2f.2l 


27.37 ■ 


33.14 


12.79 


16.65 


14.84 j 
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followup session occurred during the sixth week. At the end of the third 
week participants from whom the Weekly Log -for the preceding veek(s) had 
not been received yer e telephoned. In each case the Intervention^ 
either In the form of a phone call or the followup session^ may have had 
the effect of Increasing the self-reported frequency of activities* It 
Is not possible to determine whether this effect was the result of 
Increased participant awareness of problem solving activities^ or the 
exaggerating of frequencies In response to social pressure* 

It Is also Interesting to note the obvious differences In the 
ordering of the mean total frequency for each activity over the nine- 
week period that are dependent on the focus, of the activities. This 
suggests that different sets of skills are used In relation to dlffereht 
audiences* The. three activities reported most frequently used In rela- 
tion to students are; (a) made or got some tests or questionnaires to 
collect data about a problem^ (b) tried to make an actual statement about 
what a particular problem was» and (c) discussed particular problem 
solving procedures* On the other hand» the three activities most 
frequently used In relation to faculty and colleagues are: (a) discussed 
specific problem solving procedures^ (b) attended to problem o£ teamwork 
and group Interaction^ and (c) recommended specific problem solving 
procedures* There appeals to be a tendency to use and discuss skills 
that are dlagnostlcally and analytically oriented In relation tqj^ork^ 
with students » SUPS skills most frequently used In relation to 

work with faculty and peers are those that are more oriented to the 
generation of f^olutlons. 

Impact on Secondary Groups * The following <{uestlon was answered- by 
data gathered from the two administrations of the School Activities 
Questionnaire (SAQ) to the students of the RUPS trainees* 




Does RUPS training r0$ult in changes in classroom ctin\ate? ^ 

' This data tested the hypothesis that RtJPS training has long-term 
effects on the learning environment » The data and statistical analysis 
are reported in Table J,dJ These analyses were conducted only from those 
RUPiS trainees having complete data* 

* - T 

\ - 

Tablft 16 

Correlated t-Values 
SAQ — RVFS Treatment Group 





Pti 

X 5D 


X 5D 


Diffitftnce 

X 5D 






p-V«lui 


IUinfdtc«i€nt of 
5tIf*Conccpt 


12.00 


,61 


11,75' 


.71 


-.25 


.67 


-2.04 


Jfl 


.05 


CI«Mtooti P«tticlp«tion 


10.49 




10,32 


*fl3 


-.17 


,47 


-i.ee 


2fl 


,07 


IndlvldutUzction 
Icwttuction ^ 


11.46 


.67 


11.44 




.03 


.64 


.21 


28 


*fi4 


DCBOCt«tic ClJiitOO* 

Control 


14,19 


1*30 


14.39 


1,20 


.20' 


,fl2, 


1,31 


28 


,20 



) 

The results of the correlated t'tests indicated that the only 
significant change in any of the subscale measures over a three^onth 
period was for the reinforcement of self-concept scalft witli the 
direction of the change being negative* Based on these results* there 
is no evidence that* as measured by the SAQ» RUPS training had any 
positive long-term effects on learning environment* , 
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Do RUPS trainees involve their stwimtB in problem 
solving activities more than do teachers who have not 
received RUPS training? 

It was hypothesized tiat since RUPS is a workshop in problem 
Solving that is heavily oriented to use in classrooms* participation ' 
in RUPS training would lead to a change in problem solving behaviors 
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affecting the learnings environment and that these changes could be 
detected In student ratings of classroom climate as measured b]^^ the sAq* 

The analysis technique used to test these hypotheses was path 
analysis* The path diagram reflecting these hypotheses Is an extension 
of the path diagram presented In Figure 2 and Is presented In Figure 3* 

Variables were selected for this path analysis on the basis of 
their aat<|h with the overall conceptual model presented In Figure 1 and 
the reliabilities of the measures used* As stated earlier It was 
assumed that age, sex, class size and RUPS training (a dummy variable 
Where RUPS " 1 and control " 0) were all reliably measured* The task 

accomplishment scal e was chosen as th e mea sure of teacher attitude because - 

of Its relatively superior reliability compared to the other PSOQ scales 
(see Table 2). Level of classroom participation. Individualization of 
Instruction, and democratic classroom control were chosen as the final 
dependent variable because their posttest reliabilities were all 
significant beyond the *01 level* 

The recursive j regression equations corresponding to this path« 
diagram are as follows; 

Zg - Pg^iZi + P5^222 + n,323 + ^5,4^4 + ^-S.a^a 

h ' h.lh + ^6,222 + ^6,323 + ^6,424 + ^6,525 + h,}>h 

T;^ble 17 Includes the multiple correlation and the path coefficients 

Indicating the presence of a direct relationship between each of the 

exog^ous variables andfhe^Tevel of classroom participation* While 

the mulClpl-er-'Sofrelatlon Is not significant, the path coefficients 

(standardized partial regression coefficients) Indicate . that the data 

xonformed to the hypothesis that, teacher attitude toward problem solving 

Inflaenced classroom climate as perceived by students* However, there 
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'*BS no e.^ldence that training had any Impact on classroom climate as 
measured by the level of classroom participation* 

Table IV 

Multiple Correlation and Standardised Partial Regression 
Coefficients f6r Explaining Posttralnlng Level of 
Classroom Participation^ 



Exogenous 
Variable 


. Multiple 
Correlation 


Standardised 
Partial Regression 
Coefficient 


p-Value 


Age 


.46 


.21 


.32 


Sex 




.29 


.13 


Class Slse 




-.21 


.25 


Training 




.11 


. .54 


Posttralnlng Teacher 
Orientation Touard 
Problem Solving; 
Task Accomplistmient 




.46 


.02 



^pnly subjects for whom there were complete data were used in this 
analysis. (Classroom; N«26, RUPS; N«10, Control; N«36,, Total) . 
(Student; N-650, SUPS; N-281, Control; N-931, Total) 

Tables 18 and 19 present replications of the path analysis using 
Individualization of instruction "^and democratic classroom control as 
the finals dependent variables in the place of classroom participation. 
Essehcially similar results were found. The major difference being that 
the multiple correlation obtained in Table 19 was signl^at^t (p j< .03) 
indicating a better fit when democratic classroom control was used as a 
final dependent variable. While training had no effect upon either 
Individualization of instruction or democratic classroom control, these 
data were again consistent with the hypothesis that teacher orientation 



toward problem solving influenced student perceptions of classroom climate. 
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Table 18 



Hulciple Correlacion and Scandardized Partial Regression 
Coefficients for Explaining Fosttraining 
Level of Individualization of Instruction^ 



Exogenous 
Variable 


Multiple 
Correlation 


Standardized 
Partial Regression 
Coefficient 


p-Value 


Age 


-44 


.24 


.25 


Sex 




.17 


.35 


Class Size 




-.10 


.58 


Training 




-.19 


.31 


Posttraining Teacher 
Orientation Toward 
Problem Solving; 
Task Accomplishment 




.38 


,05 



^NOTE: Classroom: .11-26, SOPS', N-IO, Concrol; lS-36, Total 
SCudenc: 11-650, SUPSi 11-281, Control; N^JSl, Total 



Table 19 



Multiple Correlation and Standardized Partial Regression 
Coefficients for Explaining Posttraining 
Level of Democratic Classroom Control^ 



Exogenous 
Variable 


Multiple 
Correlation 


Standardized ■ 
Partial Regression 
Coefficient 


0 * 

p-Value . 


Age 


.57 - 


.24 


, .21- 


Sex 




.22 


'.20 


Class Size 




.20 


.24 


Training 




-.02 


.93 


Posttraining Teacher 
Orientation Toward 
Problem Solving; 
Task Accomplishment 




.49 


.01 



^Dt'E: Classroom; K-26, RUPSi N-l6, Control; 11-36, Total 
Student: 11-650, RUPS; 11-281, Control; 11-931, Total 



CHAPTER FOUR: CONCLUSIONS 

A summary of the findings from this study will be presented In 
this chapter* The conclusions will concern: (a) participant attitudes 
toward the RUPS Instruction system^ (b) trainee performance^ and 
(c) the impact of RUPS training on secondary audiences* 

J. 

Participant Attitudes Toward the RUPS Instructional System 

The majority of the participants* responded positively to all 
questions thac related to the workshop's success In meeting Its goals 
and objectives^ presenting relevant materials and concepts » meeting 
- personal" expectations^ provldlng~^plEf3i^tlc'al'lnfonQatloni teaching useful 
skills^ and helping participants understand. the role and possibilities 
of problem solving In schools* These positive opinions were generally 
maintained over a 3-^month period* However^ participants became 
less positive about the workshop* s success In offering new Insights 
and ways of vlewln; old problems^ fewer people responded positively 
concerning the use and applicability of RUPS to work with students » 
and fewer people summed up their workshop experience positively* The 
enthusiastic attitudes of the participants immediately following the 
workshop as reflected by their responses to the questionnaire were 
confirmed by the observations of the evaluators who noted that the 
activities and interpersonal Interactions in the workshop appeared to 
generate considerable enthusiasm on the part of the participants* 
These results 'indicate that persons who participate in RUPS 
workshops generally develop poslt;,i^e» if not enthusiastic^ attitudes 
about all aspects of the instructional system* This is most likely 
due to the Important role played by personal and group dynamics in the 
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WPS training, since' both participants and observers attributed positive 
participant attitudes to the group exercises* Furthermore, these posi- 
tive attitudes appear to remain stable over a three-month period « The 
few negative changes that occurred may be most likely attributed to 
adjustments caused by feedback from attempts to use the techniques 
taught by ^i^S* TheV most probably reflect a more realistic under- 
standing of the applicability of RUPS In the school environment » In 
contrast: to the gtoeral enthusiasm generated by the workshop* However, 
the fact that most attl^t^des remained positive appears to Indicate that 
the participants did not receive excessive negative feedback In their 
atJtemp.ts«to-employ^fli/?S^-Cechyiques* ~ — 

Trainee Attitude \ . . - . - . _ — ' ^ 

There was a significant Increase In tb^ RUPS trainees^ orientation' 
toward task related problem solylng^and classroom related problem solving* 
The RUPS group was compared to a nontrM^ment control group by means of 
path analysis* Training did not appear toshave any effect upon teacher 
attitude as measured by the task accomplishment scale of the PSOQ* 
The responses of the participants to the questions concerning 
their working understandings of the materials did not change over three 
months* At the conclusion of the workshop and again after three months, 
participants rated their working understanding of RUPS highly. Indicating 
a high degree of confidence In their abilities to understand and perform 
the skills and concepts learned In training* There Is, however, no 
other evidence of the understanding and performance of problem solving 
activities of the RUPS participants that would support these self-reports* 
The SAQs that were administered to the classes of the RVPS participants- 
which would have provided supporting evidence did not produce significantly 
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positive changes on any of the subscales* Although there Is no 
evidence available as to the actual performance change^ these results 
do provide evidence concerning the confidence and assurance of the 
participants In their ability to Implement the RVBS process. It Is 
likely that this self^report Is more an Index, of attitude and participant 
satisfaction rather than an Index of perforiitance or performance change* 

The evidence In this section tends to support the conclusion of 
the previous section; RUPS Is successful In promoting positive attitudes 
toward training In Its trainees* The positive attitudes are reflected 
In the high degree of confidence and assurance the participants express 
having In their working understanding of the materials* 

Trainee Behavior 

The results from Forms 1 and 2 of the Weekly Log Indicated that 
after training RUPS participants tended to Identify fairly complex 
problems for their improvement projects that are pursued for at least 

i ■ ' 

J 

a months and that they perceive themselves engaging In RUPS^ltkB 

problem solving activities with students and with faculty and colleagues* 

\ ■ 

As no comparative data are available on the Weekly Log». either from 
other group^ or from the RVPS participants prior to the workshop^ it is 
not .possible to determine whether these reported behaviors reflect 
changes in trainee performance^ or if they do ^ that these change? are 
attributable to RUPS training* 

In addition^ it is likely that the responses of the trainees 

r 

reflect a heightened awareness of and sensitivity to their problem 
solving behaviors due to the reactive effects of the instruments. This 
is supported by the trend in the number of activities reported weekly 
on Voxm^lt where the number peaked in the middle of the reported period 
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and fell off toward the end* This most likely Indicates that the 
participants were Initially sensitized by the Instruments » but that the 
effect wore off with repeated applications* Thus» It would appear that 
the conclusion that can be draxm from these datm Is that the self-reports 
of the participants probably reflect an Increased awareness of » and 
sensitivity to» problem solving and problem solving Jbehavlors possibly 
due to the Instruments^ but the Weekly Log does not provide evidence as 
to the existence of any changes In performance due to FlUPS training* 

Participant Behavioral Changes with Impact on Secondary Audiences 

The secondary audiences that are being considered In this study are 
the students and peers (faculty and colleagues) of the RUPS trainees* 
The behav^ral changes and the nature of the Impact were assessed through 
Form 1 of the Weekly Log and the SAQ* 

RUPS trainees report engaging In i?£/P£^-llke problem solving 
activities with both their students and peers* The frequency of these 
jictlvltles^ however^ may be somewhat Inflated due to^ the reactive 
effects of the Instruments as discussed previously! The findings 
Indicate that RUPS trainees perceive themselves as using problem solving 
skills^ but that the type of skills used differs In relation to the 
specific audience that Is being addressed* However^ since there are 
no preworkshop data or comparison groups that used the Weekly Log» It 
Is not possible to determine If changes In this type of behavior were 
cuased by participation In the RUPS workshop* 

Concerning. Impact of this behavior » it yas assumed that the self- 
reports of RUPS trainees of their use of problem solving processes and 
skills are indicative of a change In problem solving behaviors* It was 
hypothesized that these changes would In turn effect a change In the 
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learning environment US measured by the four subscales of the SAQ* 
However, student ifatlngs on the SAQ did not change significantly over 
time for the RUPS group from pretest to posttest except forgone negative 
change* reliabilities precluded definitive conclusions concerning 

the effectiveness of EVPS training In modifying classroom climate as 
measured by four subscales* In an attempt to focus the analysis on 
variables that possessed adequate Jfellablllty one classroom climate 
mea6ure> classroom participation, was used In a path analysis designed 
to reflect the complex hypothesis that RUPS training affects classroom 
climate through Its Impact upon the orientation of teachers toward 
using the problem solving procedures advocated In BUPS* The analyses 

did not support the hypothesis dealing with training effects on either 

b * 

the teacher attitude measure or the level of classroom participation as 
seen by students* However, some support was given to the hypothesis 
that teacher orientation toward problem solving was related to students^ 
perception of the level of classroom participation* When -this analysis 
was repeated for two additional climate scales. Individualization of 
Instruction and democratic classroom control, essentially similar 
results were obtained* 

Based on the data collected In this study, there Is no evidence 
of behavioral changes that Impact on secondary audiences and that may 
be attributed to RUPS training* Although the participants perceive 
themselves as using and discussing their problem solving skills with 
students and peers and exprASs a high degree of satisfaction with the 
training, there Is no evidence that would support the conclusions that 
either their behavior Is different from their behavior prior to the 
training, or If their behavior Is Indeed different. It has any 
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measurable impact on the secondary audiences In terms of the aspects of 
classroom climate measured in this study. There was some evidence thajti 
orienting behaviors of trainees as measured by the PSOQ task 
accomplishment scale did relate to the classroom learning environment* 

Summary 

The results of this study indicate that RUPS was effective in 
developing positive attitudes toward problem solving training* The study 
did not Supply evidence from which either long- or short-term performance 
changes can be inferred. The results do not necessarily indicate that 
RUPS training was not effective in promoting change* It is possible 
that a variety of shortcomings and limitations of the evaluation design 
did not allow for the identification and measurement of these changes* 
In the same sense» the lack of evidence as to the Impact of RUPS training 
on secondary audiences does not necessarily indicate that RUPS had no 
impact* This finditig may be due to the particular approach taken to 
this evaluation. Results do suggest^ however^ that teacher attitude 
toward the application of problem solving techniques in dealing with 
classroom tasks may be a viable route to modifying classroom climate* 
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Appendix A: 

STAGES OF EVALUATION AND PRODUCT 
DEVELOPMENT OF THE IMPROVING 
TEACHING COMPETENCIES PROGRAM 



The management plan for the Improving Teaching Competencies 
Program (see Resoia^ce Atto<jation ccnd Management Ptan^ 1974) divides the 
work flow for the development of an Instructional system into five 
phases: Planning* Pllot» Interim* Field Test and Outcome.. Each phase 
consists of certain development* evaluation and field relations 
activities that culminate In a milestone report. 

The range of activities associated with developing an Instructional 
syst^ Is summarized In Diagram I>. pages 81*90. These activities are 
divided Into five major categories: needs* objectives* product develop* 
ment* testing and Implementation. Diagram I also partitions these 
activltl^^s^Wong seven functional areTs including ^nagemen1£ , develop- 
ment> field relations* dissemination* formative evaluation* Internal 
summatlve evaluation and external summatlve evaluation. The matrix is 
not necessarily prescriptive nor are the evaluation relationships among 
each part strictly linear. 

^ The specific activities engaged In during the development of an 
Instructional system differ according to the phase under consideration* 
the unique needs of the specific product or change support process 
being developed and* occasionally* the style preferences among work 
unit teams. For more specific and detailed statements* reference should 
be made to the development and evaluation plans and documents for each 
work unit. 

Evaluation differs according to each phase of the development. 
During the initial phases* evaluation focuses on formative Issues and 
provides information primarily for system developers. During the 
latter phases* the emphasis is on summatlve evaluation which provides 
information and judgments for potential users of the system. This 
shift in emphasis I3 Illustrated in Figure A/ 
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Formative 

Focus 
of 

Evaluation 

Summatlve 

A A A A A 

Developmental Phases; Planning Pilot Interim Field Outcome 

Figure A* Evaluation Emphasis in the Developmental Phases* 

The following paragraphs describe in general terms the ways 
develc^pment and evaluation activities are organized for each phase of 
product, development* 

Planning Phase * In this phaae» several )cey activities provide the 
^ focua for effojrt* The initial conception^'of the proppsed instructional 

system is described along with its intended objectives* h need for the 
proposed system is documented^ and evidence provided that adequate con- 
ceptualizations and instructional strategies exist or can be developed 
feasibly for the proposed tra^ining package* Initial development » 
evaluation and dissemination plans are produced^ as are timelines/ 
staffing needs and budgets* ^ 

Pilot Phase * In this phase, a prototype of the instruction^al system 
is developed and tried out .on a small group of users from the' target 
group* Objectives of the system and entry conditions for participants 
are clarified* Program evaluators provide formative evaluation informa- 
tion to assist developers with revisions* The information includes, 
obst^rver and trainer assessments of participant involvement in the 
activities, measurements of participant satisfaction with the content, 
strategies and utility of the system* The workability of the activities, 
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the logic of the content and the quality of the teaching \lds 
materials are also assessed at this phase by the user groups/^ \ - 
Description nnd preliminary assessment of trainee outcomes ^re Initiated* 
The collection of Information regarding the marketability and costs 
of the Instructional system commences during the pilot phase as does the 
documentation of the developers* claims regarding the Intents. of the 
system It^ comparison to existing^ alternatives* 

Interim Phase * During this phase » the Instructional system goes 
through one or more cycles of revision and a nearly finished product Is 
completed* By the end of this phase^ the appropriateness of objectives 
has been determined^ statements of objectives finalized and Instrumen*^ 
tatlon to measure these selected or developed* For Instructional systems 
requiring a workshop format » specif ideations are determined for desirable 
workshop conditions and qualifications for effective tftalners^ 

The major focus of the evaluation activities fbr this phase Is on 
confirmation of the system*s ability to produce specified short-*term- 
outcomes and to test the workshop conditions » trainer qualifications 
and dissemination feasibility* This may be accomplished partially 
through conducting a ^^crlterlon workshop** designed to resemble closely 
the field-- conditions* The basic decision served by evaluation Is 
whether the Instructional system Is ready for Internal st^natlve 
evaluation and adequate for comprehensive field and outcome testing* 

Field Test Phase * In this phase^ minor revisions are made on the 
Instructional system and a product close to finished -^form Is expected 
to exist* Also» in this phase^ an Internal summatlve evaluation will 
focus on assessment of short-term outcomes of the Instructional' system* 
Specifically^ this means finding answers. to questions regarding knowledge/ 
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awareness and attitudinal growth, and participant performance change 
that can be expected as a result of active partici pat ion in the sy^ftem's 
training design under field conditions with typical trainees, trainers 
and workshop settings* Variables related to problems of installation 
and dissemination may also be examined at this point* 

Outcome Phase * During this phase, which may occur simultaneously 
with the previous phase, the instructional syst^ is finished and internal 
sunnoative evaluation will^assess the system's ability to produce, not 
only specified short'term outcomes in terms of participant satisfaction. 



knowledge, awareness or attitudinal gain and performance change, but also 
transfer, retention and impact upon secondary audiences such as students 
and/or peers* At this point evaluation plans are made for external 
summative evaluation studies such as critical comparisons between the 
outcomes of the instructional system being evaluated and outcomes produced 
by other relevant treatment efforts* External sutmnative validations of 
the product are also completed in this stage* 
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EVALUATION STUDY 
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Name 

System _ 
Trainer 

Site 

Date 



Sex 



Age 



1. 
2. 
3. 
4. 

5; 
6. 



HOME MAILING ADDRESS: Street 

City 



Phone 



State 



:ip 



WORK ADDRESS: Street 
City 



Phone 



State 



Zip 



POSITION: (Check one) Primary (1-3) Elementary (4-6) 

^ Jr. High (7.-9) Senior High (10-12) 

YEARS TEACHING EXPERIENCE: (Check appropriate space below) 

0 1-3 4-6 7-10 11 or more 

¥lGHEST DEGREE OBTAIIIED: BA MA Doctorate 

CHECK THE APPROPRUTE SPACES IF YOU HAVE BEEN -INVOLVED IN OTHER 
-NtfREL-INSTRUCTIODAI^SYTEMS : 



Interaction Analysis 

Facilitating Inquiry 

> 

Higher Level Thinking 

SAFE 

Conflict-Negotiations 



Interpersonal Influence 

Interpersonal CommunicatlonB 

Group Process Skills 

PETC-I 

PETC-II 

PETC-III 



7. 



'People attend workshops for a variety of reasons. Please ch eck 
honestly any of the following reasons that apply to you, and circle 
the checkmark of the reason which is most important In your decision 
to attend. 

^ It satisfies a requirement or gives me credits I need. 

Many others in my school were attending. 

My superiors- suggested I go. 

My superiors gave ^e the opportunity to go. 

I was selected to attend. ^ 

My attendance was paid for. 

I came because I really wanted to learn. 

I'd heard 

I had a particular problem^^£o_solve or deal with and thought 

this training would help me7~^ 
Other (what?) 
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PROBLEM SOLVING ORIENTATION QUESTIONNAIRE 



This questionnaire Is belpg used to obtain Information concerning 
various orientations toward! problem Solylng. Please answer. the 
following questions as honestly and completely as possible. The Problem 
Solving Orientation Questionnaire Is to evaluate the RUPS system, NOT 
YOU. Information collected will be used only for these evaluation 
purposes; all responses will be kept completely confidential. 

Directions for Completing the Questionnaire 

^ A separate answer sheet has been'provlded for this test. Please be 
sure to write your name in the provided space. Mark all of your responses 
on this sheet . IT IS MOST IMPORTANT THAT YOU USE ONLY A UTOMBER 2 PENCIL. 

On the scales below you will be asked to make predictions concerning 
your behavior In a variety of situations. Mark your responses In the 
way which corresponds to your predictions about your owt behavior In the 
following hypothetical situations. An example of the type of question 
you might be asked is; 

■* 

80. Suppose your principal criticises your treatment of a 
certain classroom problem and you feel the criticism is 
unjustified due to circumstances unknown to the principal. 
What are the chances that you would try to defend your 
actions by. telling the principal of these circumstances? 

1*' Almost none (less than a 10 percent chance) 

2. Maybe (about a *25 percent ohance) 

3. Possibly (about a 50 percent chance) 

4. Almost always (about a 75 percent cKance) 

5. Definitely (greater than a 90 percent chance) 

tf you think this is how you would behave in such a situation, your 
answer would be definitely (greater than a 90 percent chance) and your 
answer sheet would look like this: 

1 2 3 4 5 

80. □ D D D I 

If, on the other hand, you are not sure but think this is how you 
would pogslbly behave (about a 50 percent chance), your answer sheet 
would look like this; ^ 

1 2 3 "^4 5 

80. D ■ D ■ D D ^ 

When changing an answer be sure your first answer Is completely 
erased before darkening the cdlumn of your choice. MAKE SURE THERE 
ARE NO STRAY MABKS ON YOUR PAPER. , 
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Suppose you have a particularly difficult day with your class* 
What are the chances that ^u would spend some time listing the 
specific problems you had encountered? 

\ 1* Almost none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

Suppose some' other staff member at your school mentlones a problem 
that you are also experiencing* What are the chances that you 
would describe your own feelings or reactions to the problem? 

1* Almost none (less than a 10 percent ch nee) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chat^ce) 

Suppose some of the students In your class seem to have difficulty 
mastering a particular skill* What are the chances that you would 
consider how your own behavior was related to' theirs as you 
analyzed their problem? 



1* Almost none (less than a. 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 



Suppose you were trying out a new curriculum or project^ln your 
classroom* What are the chances that you would regularly keep 
a written record of problems you encountered? 

1* Almost none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4*-' Almost always . (^bout a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

Suppose something has gone wrong In your school — something that 
affects everyone and has everyone pretty upset* What are the 
chances that you would remain quiet and wait for others to 
analyze ttie problem? 

1* Almost none (less than a 10 percent chance) 

2f Maybe Xabout^a 25 percent chance) 

3* Possibly (about a 75 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 
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6» Supp'ose you and the students In your class have Identified a 
problem related to lunch time behavior. What are the chances 
that you would encourage a class discussion of possible ways 
to overcome the problem? 

X* , ^Almost none (less than a 10 percent chance) 

2, Maybe (about a 25 percent, chance) ^ . 

3, / Possibly (about a 5(1 percent chance) ' 

4» Almost always (^bout a 75 percent chance) 

5, Definitely (greater than a 90 percent chance) 

7» Suppose the staff Is discussing a problem related to sharing 
equipment and materials. What are the chances that you would 
malice a suggestion about a possible new arrangement that could 
alleviate the prot^emV 

1, Almost none (less thWn a 10 percent chance) 

2, M;,ybe (about a 25 percent chance) 
^ Possibly (abt5U*^...a..^ percent chance) 

4* Almost always (about a 75 percent chance) 

5, Definitely (greater than a 90 percent chanc^ 

8, ' Suppose you thought that many students In your class were not 

doing well In a,^partlcular area, What^are the chances that 
you would go to them' and ask for their suggestions ab;^ut ways 
that might Improve the situation? 

1» Almost none (less than a 10 percent chance) 

2, 'Maybe (about a 25 percent chance) 

3 , , Possibly Jabout^a 50^ P?5c?nt chance) ^ _ , 

4, f Alfno^^t always (about a 75 p*ercent chance) 

5, " Definitely (greater than a 90 percent chance) 

/ • 

9, Suppose you and your class decided to take on a long*term 
project to help .the scliool or community In s6me way, Vfhat 
are the chances tha^ you ^nd-your class would jointly plan 
all the steps Involved and consider how you would monitor 
progress as yop Implemented the project? 

^ 1, 'Almost none (less than a 10 percent, chance) 

2, ' Kaybe (about a 25 percent chance) 

3, Possibly '(about a 50 percent chance) 

4, Almost- always (about a 75 percent cKance) 

5, Definitely (greater than a 90 percent chance) 

10, Suppose ypu set a goal to Improve Instruction In a particular 
area. What are* the -chances that ypu would make a detailed 
written plan Including a list of forces that could restrain 
Implementation before starting on the project? 

1, Almost none'(less than a 10 percent chance) 

2, Maybe (about a 25 percent chance) 

3, Possibly (about a 50 percent chance) 

4, Almost ^always (about a 75 percent chance) 

5, Definitely (greater than a 90 percent chance) 

. . Si • . ' " 



ll* Suppose you have an exciting Idea for a new way of teaching that 
could really enhance the degree of learning of students, but 
suppose also that there Is a possibility that the new approach 
might not 90 over at all* What are the chances that you would 
try out the new method anyway If there was a high risk that the' 
new method would fall? 

1* Almost none (less than a 10 percent chance) 

2* Maybe (abouc a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

Suppose you develop a particularly useful and effective method for 
teaching something* What are the chances that you would: 
(Questions 12 and 13) 

12* Describe It briefly at a faculty meeting and offer to meet with 
*others who want to hear more about It? 

^ ^ 1* Almost none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

♦ 

13* Say nothing about It unless somebody asked ycu? 

1* 1 Almost none (less than a 10 percent chance) 

2* M^ybe (about a 25 percent chance) 

' 3* Possibly (about a 50 percent chance) 

4* Almost always (abo'ut a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

Suppose you wanted' to Improve your classroom effectiveness In some 
area* What are the chances that you would; (Questions 14, 15 and 16) 

14* Ask another teacher to observe your teaching and then have a 
conference with you afterwards? ^ 

1* Almost none (less than a 10 percent chance) 
2* Maybe (about a 25 percent chance) 
3» Possibly (about a 50 percent chance) 
4* Almost always (about a 75 percent chance) 
5* Definitely (greater than a 90 percent chance) 

* 

15* Ask another teacher to let you observe how he/siie teaches In 
order to get an Idea how to Improve your own teaching methods 
and effectiveness? 

* 

1» Almost-none (less ttian a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3» Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 



XS\ tJse a questionnaire to find out how your students^ feel about 
your teaching In so|ae area? 



1« Almost none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance)^ 

3* Possibly (about a 50 percent chance) 

4» AJ.most always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 

Suppose you are present when two other teachers get Into a heated 
argument about how the school should be run» What are the chances 
that you Would; (Questions 17» 18» 19» and 20) 

17* Listen to both parties In the argument and then side with the 
one you think Is right? 

1* Almost none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

^4* Almost always (about a 75 percent chance) 

5» Definitely (greater than a 90 percent chance) 

18* Avoid getting Involved In. the Interaction at all? 

1* Almost'none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Mmost always (about a 75 percent chance) 

5. Definitely (greater than a 90 percetifc chance) 

19* Help each one In the argument to understand the viewpoint of 
the other? 

1* Almost none (less than a 10 percent chance) 

2* Maybe (about a 25 percent chance) 

3* Pos^slbly (at)out a 50 percent chance) 

4*. Almost^always (about a 75 percent chdnce) 

5* Peflnltely (greater than a 90 percent chance) 

20* Try to get the two to quiet doxm and stop arguing? 

1* Almost none (less than a 10 percert ^chance) 

' * 2* Maybe (about a 25 percent chance). 

3* Possibly (about a 50 percent chance) ^ ^ 

4* Almost always (about a 75 percent chance) 

5. Definitely (greater than a 90 pertfent chance) 
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E* Suppose you strongly disagree with a procedure that the principal has ^ 
outlined for all to follow* What are the chances that ^ou would: ^ 
' (Questions 21, 22, and 23) , 



21*' Go and talk with the principal about this disagreement? 



1* Almost none (less than a 10 percent chance) 

2* Haybe (about a 25 percent chance) 

3* Possibly (about a 50 percent chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 



1* Almost none (less than a 10 percent chance) 

2* Haybe (about a 2S percent chance) 

3* Possibly (about a* 50 p'ercenL chance) 

4* Almost always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 



23* Say nothing but comply with the principal's directive? 



1* Almost none (less than a 10 percent chance) ^ 

2*- Maybe (about a 25 percent -chance) 

3* Possibly (about a 50 percent chance) 

4* Almost '-always (about a 75 percent chance) 

5* Definitely (greater than a 90 percent chance) 



F* One^of the frustrations in working in schools is that things don^t 
always go as well as planned* Suppose something has gone wrong in 
your school—something that affects everyone and has everyone pretty 
upset* What are the chances that you would: -(Questions 24, 25 and 26) . 

'24* Hake suggestions to the staff regarding^who was responsible for 

things going wrong so accountability can be established the next . ; 
time things go wrong? 

1* Almost none (less than a 10 percent chance) 

-J 2* Haybe (about a 25 percent chance) 

^ "3* — PoTinlhly (nhnnr a "^Q pf ^cent chance) 

4s' Almost always (about a 75 percent chance) ' 
5* Definitely (greater^ than a 90 percent chance) 

25* Remain quiet and wait for others to analyze the problem? 



< 



22* 



Say nothing buc ignore the principal's directive? 



1* 
2* 
3* 
4* 
5* 



Almost none (less than a 10 percent chance) 
Maybe (about a 25 'percent chance) 
Possibly (about a 50 percent chance) 
Almost always (about a 75 percent chance) 
Definitely (greater' than a 90 percent chance) 
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26. 'Make suggestions' to the staff about possible nei^ arrangements 
to present this upsetting occurrence? 

^ 1. Almos^t none (less than a lO percent chance) 

2* Maybe (about a 25 percent chance) 

^. Possibly (about a SO percent chance) 

.4. Almost always (about d 75 percent chance) 

S. definitely (greater than a 90 percent chance) 

Suppose you have worked .hard In developlni? a aew way for teaching 
something to your students. However, six months later you discover 
that achievement had not Increased. In fact, several students are 
doing less well tliai) before. What are the chances that you would: 
(Questions 27,c28, 29, 30, and 31) 

27. Go and talk to the principal about what you had done?. 



1* Almost none (less than a 10 percent chance) 

2. Maybe (abdut a 25 percent chance) 

3. Possibly (about a 30 percent chance) 

4. Almost^always (about a 75 percent chance) 

5. Definitely (greater than a 90 percent chance) 

If 



28. Say nothing but start working out another plan? 

1* Almost n^ne (less than a 10 percent chance)"^ 

2. Haybe (about a 25 "percent chance) 

3. Possibly (ab ^ut a SO percent chance) 

4. Almos^ always (about a 75 percent chance) 

5. De'flnltely (greater than a 90 -percent chance) 

29. Go and talk to another colleague about what you had done? 

/ 1. Almost none (less than^a lO percent chance)- 

2. Maybe (about a 25 percent chance) 

3. Poss,lbly (about a 50 percent chance) 

4. Almost always (about a 75 percent chancf^) 

5. Definitely (greater than a 90 percent chance) 



^0. vpry fH^app^lnt^^l t \ \v\ fffi i^ -TfUi ft^*n?y of ;allure? 



1. Almost none (less than 10 percent chance) 

2. Maybe (about a 25 percent chance) 

3. Possibly (about a SO percent chance) 

4. Almost always (about a 75 percent chance) 

5. .Definitely (greater than a 90 percent chance) 

31. Hake a list to describe What^s ^olng well and what's geUlng 
, In the way before you started to revise your t>lan or develop 
a new one? 

1. Almost nolle (less th&n a lO percent chance) ^ ' 

2. ^ybe (about a 25 percent chance) 

3. Possibly (about a 50 percent chance) 

4. Almost always . (about a 75 percent chance) 

5. Definitely (greater than a -90 percent chance) 
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SCHOOL ACTIVITIES QUESTIONNAIRE 



Directions * 

The purpose of the questions In this booklet Is to find out vhat 
your class is like* This Is not a "test/* Your teacher will not see 
your answers and you do not have to put -your name on the answer shee^t* 

This Is an example of the kind of questions you might be asked: 

80* Do you like to come tu school? 

1* Yes , 
2* No ' 
3* Sometimes 

To answer this question, first decide If your answer Is "Yes/* 
*'No," or "Sometimes*'* If your answer Is **Yes,** look at your answer 
sheet (the blue and white paper) and find Question 80* With your 
pencil darken Column 1 of Question 80* 

Your answer on the answer sheet would look like this: 
* 

1 2 3 4 5 
80. I □ D □ □ 

If your answer >wa8 ''Sometimes,*' Number 3, your answer would look 
^llke this: " ' • ' 

1 2 ■ 3 4 . 5 

80. Q □ ■ □ 0 

If you want to change an answer, be sure to erase your first 
answer and darken the column for your. final answer* 

Work as quickly as you can* The person administering the test 
will tell you when to stop* 

PLEASE TRY TO GIVE YOUR HONEST FEELINGS ABOUT YOUR CUSS* 



Do you fike to come to school? 



1. Yes 

i. Ko 

3* Sometimes 

2. Are you proud of the things you do in school? 

1. Very proud 

2« Proud of 8ome things » not proud of others 

3. Not very proud 

3. How often do you talk to a. teacher by yourself about your school 
work? 

1/ Two or three times a day 

2. About once a day 

3. About once a week 

4. Almost never 

4. How often do thd students in your class talk to the teacher about 
how much time they should spend on an activity? 

1* Hoxe than once a day 

2. About once a day j 

3. Two or three times 3 week ^ ^ 

4. Once a week 

5. Not very often 

5. How do you usually feel wheil your teacher talks to you about your 
school work? 

1. ' Encouraged 

2. Don't know 

3. A little discouraged 

6. In the classroom^ the teacher usually calls on; 

1. The same group of students 

2. 'Almost all' the students 

7i In general^ how are problems usually solved in your classroom? 

1. Our teacher solves the problems alone 
^ 2. The teacher and the students work together 

8. Do you have activities where the teacher has you tell someone else 
about something? 

1. No» haven^t done that yet 

2. Not Very often 

3. About once a week 

4. Ye8» two or three times a week or more 
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9* Does your teach :r know vhat is eas/ and what is hard for you? 

1. No, not very veil 
2* Sometimes 
3* Yes, knows very well 

10* Who decides what the class will do? 

1* ThC*'teacher usually decides by him/herself what the class 

^-^ilT do 

2* We often plan with the teachers what we will do 

11* Does your claas have activities where many students get called on? 

1* No» haven't done that yet 

2* Not very often 

3* About once a week 

4- Yes» about once a day or more^ 

12* How often does your teacher encourage you to try a different task^ 

c 

1* Almost never 

2* Sometimes » once a week or less 

3* Fairly often^ two or three times a week 

4* .About once a day 

5* ' Two or three times a day 

13* Does your class have discussions about how the students should act? 

1* Yes 

2* Not very often 

3* No» generally the teacher tells us 

14* How often do you spend more time on some activities than other 
students do? 

1* Fairly dften^ two or three times a week'-^ 
2* Sometimes, about once a week or less 
3* Almost nev^ 

15* Do you ever work on something that other students in your class 
are not working on? 

1. No, usually we work on the same thing 

2. Sometimes /about once a week or less 

3. Fairly often^ two or three times a week 

16. How often does your teacher let students lecide how an activity ' 
or project s1;iould. be done? 

1* Almost never ' 

2. Sometimes 

3* Most of the time 
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17* How often does your teacher permit a lot of talking and activities 
In your^ classroom? 

1* A- number of times a day 
2* About once a day 

The classroom Is usually rather quiet 

IS* How often do other students In your class tell you that you have 
done a good job? 

1* Not very ,of ten 

2* About once a week 

3* About two or three times a week 

4* Once a day or more 

19* Do you think your teacher knows yhan kinds of activities you like 
the most? 

1* Not very well 
2* I don^t know 
3* Yes 

20* How often can you speak out In a classroom discussion when you 
want to? ^ 

1* Almost never ^ 
2* Not vety often 
3* Sometimes 
4* Fairly often 
5^ Always 

21* How of (en do you spend less time on some activities than other 
student^ do? 

r 

1* Fairly often^ two or three times a week 
2* Sometimes^ about ^nce a week or less 
3* Almost nevet 

22* How often do you have class aj:tlvltles where many students^take 
turns speaking? 

* 

1* More than once a day 
2* Once a day 

3* Two or three times a week 
4* About once a week 
5* Not very often 
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23. How often does^^your teacher tell you about something you have 
done well? 

1* Almost never 

2* Sometimes » once a week or less 

3. Fairly often^ tvo or three times a week 
4* About once a day 

5* Tvo or three times a day 

24* How often dp you have class discussions where many students have 
something to say? ' ^ 

1* Haven^t done that yet 

2. Not vety often 

3* About once a week 

4. Two or three times a week 
5* About once a day, or more 




10 3 



107 



Position 
Trainer 



^RUPS POSTTRAINING QUESTIONNAIRE 



The purpose of this questlonna{,re Is to provide Information ^bout 
your views concerning the RUPS workshop* The Information will be used 
to help us learn more about how workshop participants feel about che 
RUPS system* The questlonhalre Is Intended to evaluate the workshop, 
NOT YOU* Please %nsver all of the following questions as honestly and 
completely as possible; all responses will be kept confidential* 

A» The following questions (1-11) ask for your judgmenjis about ^he 
success of the workshop In achieving Its goals » Ualng the S^polnt 
scale below please circle the number Indicating how successful you 
would say the Information, materials, practice exercifies, atvd 
methods used In this workshop were In achieving tlie followlii^ goals » 



Not at all 
successful 



2 

Barely 
successful 



Somewhat 
successful 



Successful 



ExC^emely 
successful 



Not at all 
successful 



Kpctreniely 
ai ^cesaful 



How successful do you feel 
(he workshop was In: 

l/;Provldlng clear Informa-^ 
'tlon concerning directions 
and rationales for the 
different sessions * 

2* Offering new Insights, ^ 
new ways of viewing old 
problems* 

3* Addressing what you 
thought were Important 
Issues/vital concerns* 

4* Demanding original 

thinking on your part* 

5* , Presenting clearly under- 
standable definitions and 
descriptions of problem 
solving* 



1 2 
1 2 



4 



5 
5 
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Noc ac all 
successful 



Excremely 
successful 



6, Helping you underscand che 
role and possibillcy of 
problem solving In schools. 

7. Maincalning your inceresc 
chroughouc che workshop^ 

S. Providing useful skills and 
concepcs for working wich 
ochers oucside your 
profesfsianal life. ' 

9. Providing informacion wich 
praccical applicacion for 
""your work wich scudenca. 

10. ^Providing information wich 
^ practical applicacion for 
your work wich ceachers* 

XI. Providing informacion'with 
pract;;ical application for ' 
your work wich superiors. 



3 
3 



5 ^ 



5 • 



B. The following quescions (12-23) are concerned wich your underscanding 
of the concepcual materials and your aLbillty to apply the skills 
learned in chis workshop. For each of the following concepcs and 
skills, we would like you co respond to the quescion *'How well do 
you feel you have a working underscanding 'of . . .'Von che following 
scale* 



I 2 

I don't understand the 1 don^t understand th« 
concept and could not concept but could 
apply th« akilla apply th« akilla 



I und«ratand th« 
canc«pt but could 
not apply th« akilla 



I understand the 
concept and" could 
apply th« akilla 



How well do you feel you have a 
working underscanding of: 

12 < Applying che 4-guideline 
criceria for writing a 
problem scacemen'c. 

13. Using che forcefield 
diagnoscic technique. 

14* Selecting and creating 
inscrumencs for data 
gachering. 
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1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



15. Using- criteria for ^ 
deriving Implications 
from research findings. 

16. Spotting and analyzing 
major results of data 
collected. 

17. Brainstorming action 
alternatives to meet' 
implications derived 
from findings. 

18. Applying guldellnfts for 
planning and implementing 
action alternatives. 

19. Evaluating solution plans. 

20. Paraphrasing interpersonal 
communications. 

*. 

21. Using concepts and skills 
of glvl^ng and receiving 
feedback. 

22. Diagnosing teamwork^ , 
relations. 

23. ' Identifying and evaluating 

small group dynamics. 

The following^ questions deal^^ith how^you feel about 
as a whole and about its usefulness for yourself and 



1 
1 



4 
4 



24. How successful do you 
feel this workshop was 
in meeting your expecta- 
tions about what you 
personally wanted^ to get 
out of it? 0 

25. How clearly did you 

^ understand the workshop's 
overall objectives? 

26. How successful do you 
' feel the workshop was 

in achieving its overall 
objectives? 



Not at all 
Successful 



the workshop 
ottiiers* 

Ektremely 
, successful 



Very clearly 



Extremely 
successful 



Not at all 



Not at all 
successful 
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27* How useful do you feel the 
skills and concepts you 
have learned are in 
problm aolving? 

28. In all honesty » how much 
,do you plan to use the 
ldeas» skills and/or 
materials presented in 
this workshop as an 
integral part of your work? 

29. Now that the workshop is 
over /how would you sum yp 
the experience? 



30 1 Would^you reconunend this, 
workahop to a friend 
whose interests are like 
yours? 



Very useful 



Of no use 



Completely 



Extremely 
worthwhile 



Definitely 
recommend 



Not at all 



Of no worth 
at all 



. Definitely 
not reconnend 
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WEEKLY LOG 



In order that we may be able Co Judge the ef f ectlvenj&ss of the RU^S 
training system, we would like you Co keep a weekly log of whatever 
problem solving activities you engine In. Our Interest Is'* In evaluating 
the system and not your performance. Thus, w6 would appreciate your 
candid responses. There Is no need to put your name on the forms and 
all Information will be held strictly confidential. 

On each Friday, from October 25 through December 20, we wouldvllke 
you to fill out a brief 14-ltem checklist Indicating the frequency with 
whxch you engaged In certain problem solving activities during the last 
* week. This checklist Is labeled Form 1 and you will find a sample 
attached to these directions. ' You will note chat the date ^ou are Co 
recurn the form Is Indicated at the cop. On each Friday you would 
simply find the appropriately dated form. Indicate the number of times 
you engaged In each activity In. the last week, and return the form to us. 

On every other week we ask chat you fill out an additional form» 
Form 2 » jconcernlng the nature of the problems you have been working on 
during the preceding two weeks. Each problem you' work on would require 
an additional Form 2, and since you may work on several problems during 

'a. 2-week period, we have included extra forms In the back of the 
folder. I^orm 2 asks you to briefly describe che nature of the problem 
you worked on and quickly Indicate what you have done on the problem, ' 

J and what difficulties and successes you have encountered. A sample 
copy of jForm 2 Is attached. ^ . ^ • 

We have provided stamped return. envelopes In the back of the folder 
along with extra copies of Form 1 and Form 2. Every Friday, then, you 
should complete tl a appropriately dated forms and mall them to^us. We 
anticipate that It should take no longer than 15 to 3Q minutes to com- 
plete the forms and drop them In the mall. We. realize, however, that ^ 
some Fridays you may feel you, do not have che time. We would strongly 
^ urge you, however, to take the few minutes to complete^ the forms. If 
you wait several days, you will probably find It much more difficult - 
to recall your past problem solving activities. Because Ic- Is very 
Important for' us to receive these forms .on time, we will contact you 
by phone In the^^event that your forms become several days late. 

\ We greatly appreciate your taking the time to keep this Weekly Log 

for us. *lf you have difficulties or questions concerning this Log or 
any other aspect of the RUFS traln^ng^ and data collection, please feel - 
free to call one of the Individuals below collect; 

Name: ' Nick Smith Rachel Rasseja " / 

Office Phone: - (503)224-3650, ext. 323 (503)224-3650, ext. 267^ , 
' Home Fhone: (503)649-4503 ^ (503)234-8891 
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FOtiH I 



PlcAse, conpltte «nd mall this Form on Ftlday, 



In the tiucet provided to, the right, indicate 
how ttiny ti»ea in the laat week you engaged in 
the foliovlng 'activities that focuaed on 
claearooii etudente or your faculty and 
colle^guee. 



OFFICE USE 

Form LU 
1 

Week I 11 

NO, I I I I 
4 5 6 



Focua of Activity 



Activity , 

I*. Tried to laake an actual statenent of what 
a particular problem waa* 

a. Tried to id^^ntify pottenti^I difficulties and 
aide concerning a particular problem. 

3* Made or got s^e teats or queationnairea to 
collect data about e problem* 

4. Collected and analyzed data and thought about ■ 
how the 'reaulta applied to a particular problem* 

5* Thought of or Ilated different aolutione and 
plana pf action to solve a problem* ' 

6. Tried out poaaible aolutiona and plans of 
action to solve a problem* 

7. Evaluated or revised solution plans for 
solving a- problma* 

8* Discussed general problem solving strategies* 

* 

9* Discussed specific problem ^solving procedures* 

10* Suggested general problem aolving atrategiea* 

II. Recoomende^ apecified problem solving 
proceduAa* ' 

IZ* Attended to problem of feedback and interpersonal 
coominications* ^ 

13* Attended to problems of teamwork and gFpup 
interactions* 

14. Thought^ about or engaged in specific problem . 
aolving activities* 



Claaaroom 

Students 



l: 



A 



LA 



Faculty and 
CtolleaKues 



L 



-L 



U 



J 
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FORM 2 



PI^Be comp^Iece and mall ch Is, Form on Fclday, 



PIeas« cecail a pacclculdC pcobira you vocked In ch« lasc cwo 
weeks and answec ch« following qujesclons wlch Ic In vlnd* ^ 
foe each pcoblea you wcked on, complace a sepacace Form 2 
Indlcaclng cha dace as above. 



Bclefly Indictee che nacuce of che ^coblam you wocked on. 



/i ' . ■■' ' 
Was chls che pcoblem you Indlcaced ac che yes no 

wockshop meeclng chac you would be uocklng (I) (0) 

on? (check one) ' - . 

yes no 

Did you cepcfrc on chls pcoblem on a Form 2 (I) (0) 

cwo weeks ago? (check one) * ' . 

Check (/) which of che foIIoOiitg you hav« done on chls problem. 

. - (I) ^ ^ . 

» * ^ - 

I. fcled CO make ifn accual scacemenc o^ whac a 

pacclculac pcoblem was. 

Tcled CO Idenclfy pocenclal dlfflculcles and 

aids concerning a pacclculac pcoblem. 

3. ^ ^ Made oc goc some casks oc quesclonnalces co 

* collecc data abouc a pcoblem. 

4. ' CoIIecced and analyzed daca" and choughc &bouc 

' how che cesulcs applied co a>.pactlculac pcoblem. 

5. * Though c of oc IlBCtd dlffecenc soluclons and 

plans of acclon co solve i pcoblem. ^ 

6. [ Tcltd ouC 'possible soluclons ^nd plans of * 

accloa CO solve a^ pcoblem* ^ 

' ■• * 

7. £valuaced oc cevlsed soluclpn plans foe solving 

a pcoblem. 

• ^ 

Of che acclvlcles you have, checked, which one did you have che 
.Qosc^ dlf flculcy wlch? 

If you had dlfflculcy wlch anochec acclvlcy^ which one? 



How many houcs have you spenc wocklng on chls pcpblam? 



How )fiany houcs ( 



) ovec how many weeks ( ) will you 



pcobably need co cotnplece che cesc of your wock? 



OFFICE USE 
Form 

I 

Week 1 I I 

I 3 

No- 1111 

4 5 6 



9 
10 

LJ 
11 



12 



13 

u 

14 



15 



16 



a? 

I I I 1^ 
18 19 20 

I 11 1 
21 22 23 

I I I 

24 25 



FORM 2 (continued) 

Check (/) below chot« difficulties you have had In working 

(1) 

on this problem. 



1. 


I 


an experiencing retlscance fron tcudenct. 


2. 


I 


am experiencing reslscaace from faculty* 


3. 


I 


lack sufficient time to work on the problm* 


4. 


I 


lack sufficient r-eiources to work ot^ the problem. 


5. 


I 


do not have admlalfltratlve support* 


6* 


I 


do not have support from other teachers* 



Check (/) bow successful you. feel you are In dealing with this 
(1) 

part4^ular problem^ \ 
ir I an (check one): 

(1) . eatb^slastlc 

(2) ^^ -hopefu] ^_ 

(3) doubtful 

(4) discouraged 

2. I think I will be able to splve (check one)(l) all, 

(2) part, (3) none of the the problem. 

3. I will probably need (check one) (1) much, 

. (2) some, (3) tto additional help or training 

to solve this problem* 

■ 4. If you atteoded the workshop training, how^ useful do you 
feel the training was In working on this problem? 
"(check one) > 

(1) _„ very useful 

(2) moderately useful 

(3) tmuseful 

(4) hampered my efforts 
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FOLLpWUP QCESTIONNAIRE 



The purpf^ose of this queatlonnalre is to obtain 
Itifomatlon about your feallngs concerning the 
worktop 'you attended three monthe ago* Pleaae 
answ^r^all of the following queationa aa honeatly 
and completely aa posaible* All responaea will 
be kept completely confidential* 



The following queationa (32-36) aak for your judgnents about the succesa of the 
workahop in achieifing'its goala* Uaiug the^S-point acale beldw pleaae circle the 
number indicating how aucceaaful you wouI4 aay tha information, materlala, practice 
exerciaes, and methoda uae4 in this workshop were in achieving the following goals* 



Hot at all 
successful 



2' ' 
Barely 
successful 



Somewhat 
successful 



Succeasful 



Extremely 
aucceaaful 



(tow succesaful do you feel the workshop 
was in: 

jTl Offering new insights, new ways of 
viewing old problems^ ^ 

33. Addressing important issues, vital 
concerns* 

34. Presenting a clearly understandable 
description of problem aolvlng. 

35. Providing useful skills and concepts 
for working with others outside ^our 
professional life* 

36. Providing information with practical 
application for your work with 
studenta* 

37. Providing information with practical 
application for your work with , 
teachers. ^ 

36.^ Providing information with practical 
application for your workwith 
superiors* 

'The following questions (39-45) deal with how you feel about the workshop as 
a whole and about itis usefulness for yourself and others. 

39. Compared to other uorkahops you have attended* how successful do you feel 
this workahop waa^ in meeting your expectations aboUt what you personally 
wanted to get out of it? 



Hot at all 
successful 



Extremely 
successful 



Extremely 
successful in 
meeting my 
expectations 



Relatively 
succeaaful 



3' 
Neutral 



Relatively 
unaucceaaful 



Not at all 
auccessful in 

meeting my 
^ expectationa 
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40. How successful do you feel the workihop was In achieving its overall 
objectivfis? 



Excrenely 
succeiiful 



Relatively 
iucceiiful 



Neutral 



Relatively 
uoiucceiiful 



Not at all 
aucceeiful 



41* Compared to wKat you learned in oth^r workshops^ how useful do you feel 
the skills and concepts you have learned are in problem solving? 



Very useful 



Relatively 
useful 



Neutral. 



Relatively 
. useless 



Useless 



42. Now that the workshop is over» how would you sun up the experience? 



Extrcnely 
worthwhile 



2 

Very 
worthwhile 



4 

Barely . 
worthwhile 



Of no worth 
at all 



43. Would you 



Worthwhile 

recommend this workshop to any of your friands or colleagues? 



Definitely 
recoQMnd 



Possibly . 
recofwend 



Recommend with 
reservations 



Probably not 
recoOTcnd 



1>efinite^ not 
recoqsttnd 



44. In relation to your expenditure of time and/or money, how valuable has 
the workshop been to you? • ^ 



Extremely 
valuable 



Hbderately 
valuable 



3 > 
Of some value 



Of limited 
value 



Of no value 



45^^ Compareci to other workshops^ how useful do you think ^he workshop would 
be as a training experience for your^'col leagues? 



1. 



I 

Very 
useful 



Relatively 
useful 



Neutral 



Relatively 
useless 



Useless 



The following i^uestions (46-5?) .sre concerned with your understanding of the 
conceptual materials and your ability to apply the skills learned in thla workshop. 
For each of the following concepts and skills, we would like you to respon4 to the 
question^ **How well do you feel you have a working understanding of*..** on the 
following scale* 

1 ^ 2 \ 3 \ ^ 4 

' I don't understand the I don*t understand the I understand the I und# rs tend v the 
concept and could not concept but could concept but could concept and could 
'apply the SKills apply the skills not apply the skills apply the skills 



How well do you feel you hava a working 
understanding of; 

46. Applying the 4-gui(leline criteria for 
writing a problem statement. 

47. Using the forcefiald diagnostic technii^ue. 

4d» Selecting and creating instruments for 
data gathering. 

V 

49. Using criteria for deriving implications 
from research findings* 
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50* Spotting and analyzing maj^jor rafult8 
of data collaQted* 

51* Bral»toTmlng action alternatives to 
meat Impllcatloiiv darlvad frbu 
findings* 

52* Applying guldallnas for planning and 
Ijoplinentlng action aXtematlvas* 

53. Evaluatlng'aoiutlon plans* 

54* Paraphrasing intsrpersonal coaununlcatlon^ 

55* Using concepts and skills of giving and 
rac«lving fssdback* « 

$6. Diagnosing taanwork relattoiiSL.r 

57. Identifying and evaluating small gcoup 
dyoaadcs. : 



PSYCHOMETRIC EVALUATION OF THE 
PROBLEM SOLVING ORIENTATION . 
QUESTIONNAIRE ^ 



This Is a technical note designed to present psychometric ^ 

Information vhlch may be used In evaluating a self^report attitude 

Instrument called the Problem Solving Orientation Questionnaire (FSOQ) . 

The PSOQ was developed specifically for the evaluation of Research 

Utilizing ?voblem Solving (RUPSl, an Instructional system designed for 

training teachers In a generic problem solving model for classroom 

application* The PSOQ Is a self-report questionnaire designed to 

measure certain aspects of the orientation of teachers toward problem 

solving* Respondents are presented with a vai^lety of realistic probl^ 

situations and asked to estimate the probability of their reacting In 

a manner consistent with the "action research** orientation toward 

problem solving advocated In RUPS train ing* Re sponses t o Items are 

made on a 5-point multiple choice scale Identical to that In the 

following sample ltem< _ 

Suppose something has gone wrong In your school—* 
something that affects everyone and has everyone 
upset* What are the chances that you would remain 
quiet and wait for others to analyze the probl«Q? 

1* Almost none (less than a 10 percent chance) 
2* Maybe (about a 25 percent chance) 
3* Possibly (about a 50 percent chance) 

Almost always (about a 75 percent chance) 
5* Deflnltei^ (greater than a 90 percent chance) 

The items from the PSOQ were categorized according to two classl- 
ficatlon schemes determined by the contept of the 'situation presented 
in the item stem* ^e first classification scheme divides the situations 
described in the PSOQ lii^o two categories: (a) situations'dealing 
primarily with Interpersonal relationships , and (b) situations dealing 
primarily with taskr accomplishment * 

Th^ second classification schane also divides the situations 

described in the PSOQ into two categories: (a) situations related to 

* 
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classroom problan solving activities, and (b) situations delated to 
nonclassroom problem solving activities. Scores for'each set of Items / 
were the sum of the responses to each lt€lQ within that^'category* 

Two developers of the SUPS Instructional system and four evaluators 
who had never> received RUPS training were Instrumental In assigning the 
Items to the categories described above* Each developer and evaluator 
received a complete FSOQ, with each Item printed on an Index card* l^ey 
were Instructed to work Independently and assign Items either to a task 
orientation category or to an Interpersonal orientation category* Items 
which the evaluators and developers could not unanimously assign to these 
two categories wt^re not scored* Of the 31 Items In the FSOQ, only two 
.were eliminated because of lack of unanimity* Twelve Items were classified 
«s dealing with tasks'and seventeen with Interpersonal relationships* 

The same procedure of sorting cards was followed for the classroom 
and nonclassroom ca'tegorles* Twelve Items were placed In the classroom 
situation category, seventeen Items were placed In the nonclassroom 
situation category and two Items were deleted due to lack of unanimity* 

Fretralnlng data from a sample of 87 teachers who responded to the^'^ 

\* 

FSOQ were ased to calculate split half reliabilities, which were '\ 

\ . 

corrected with the Spearman Brown Frophecy formula* These 87 teachers^^^^^ 
were participants In RUPS^ tw other NUREL workshops, or^meta2>ers of the 
control group used In an evaluation st;udy* ^ ' ' 

In addition Co the spllt->half reliabilities, test-retest reliabilities 
were computed with the same group of 87 teachers* Since the test->retest 
reliabilities were basei upon pretralnlng and posttralning administrations 
designed to assess training effects, the test*rete8t relaibllitled ^should 
be Interpreted as conservative estimates* i 
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Because both pretest and posttest Information were available for 
analysis, the data were analyzed In a multltraltmultloccaslon matrix 
of Intercorrelatlons* Two tables are required as the four scales 
employed were not mutually exclusive with respect to Items. . Categories 
within classification schemes were mutually exclusive^ however^ and ^ 
Table A Includes the Inter correlations for the task accomplishment 
scale and the Interpersonal relatlonshlps'scale. A multltralt-* 
multloccaslon matrix Is used because It provides two Indices 6f scale 
^reliability (split-half and test-retest) and concurrent and predictive 
Interscale relationship all In one table> It» therefore^ provides an 
overview of a variety of scale characteristics In a single table. 

Table A 

Multltralt^Multloccaslon Intercorrelatlons ^ 
of Task Accomplishment and Interpersonal Relationships 

PSOQ Scales* 





Pre 


Post 


Task ^ Interpersonal 
Accoapliahment Relat ionshii^a 


Task ^ Interpersonal 
Accomplishment Relationahip^ 


Pre 


Task 

Ac coop 1 1 dhment 


.'55 .37* 


* 


Intfttperaonal 
Relationships 


Poat 


ti8k 

Accooplishntnt 


.Bf .40 
.34 ..H^ 


.54 — ■ 


Interpersonal 
Relationship! 



^Corrected split half fel^abilitias. 
^test*rSte9t reliabilities. 



The Italicized correlations are reliability coefficients* Those 
reliability coefficients In the pretest portion of the matrix are split- 
half reliabilities with corrections made according to the Spearmen 
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Brown' Prophecy formula. Similar reliabilities were not calculated for 
the postteat administration. Test-retest reliabilities ar& placed in - 
the cross occasion (i.e.» pre-post) portion of the matrix. The task 
accomplishment scale was more reliable than the interpersonal relation- 
ships scale indicating that it may have greater potential value for 
evaluation purposes. Within occasion the interpersonal relationships 
scale correlated less with Itself (i.e., split-half reliability) than 
it did with the task accomplishment scale. To have much confidence in 
a scale the reliabilities should be higher than Interscale correlations. 
Therefore^ further doubt is cast on the value of the Interpersonal 
relationships scale. ^ 

The pretralning and .posttraluing Interscale and wlthin-occaslon 
correlations were very nearly identical. Task accomplishment correlated 
.55 for the pretralning data with interpersonal relationships and '5^ ^ 
for the posttralning data. This pair of concurrent Interscale correlations 
Indicate a form of stability in the relationships between the scales. A$ 
expec'te4» these interscale and wlthin-occaslon correlations were higher 
than the interscale and between-occaslon correlations which were '34 
and .40. ^ 

Tabl^ B Includes the Intercorrelatlons for the classroom situations 
and nonclassroom situations .scales. The classroom sitjjations scale 
possessed greai:er reliability than th^ nonclassroom situations scale. 
While the rellabil4,ty of the nonclassroom situations scale .was higher 
than this scale's Intercorrelatlon with*the classroom situations scale/ 
the reliabillties^rjeL.not high in absolute terms.' , ' 



Table B 



Multitrait*Multioccasion Inter cor relations 
of Classroom Situation and Nonclassroom 
Situation PSOQ Scaled^ 





Prt 


.^tt 


Title InttTptrtoail 
AcconDllttmttkt RtlttlonthlDt 


Titk Xnttrptrabntl 
AccottpjLlihoitnt Rtlit lonthlpt 


Prt 


Tttk^ 

AcccuplltbiMnt 


.6^ .32 




Inttrptrtontl 
Rtlttlonthlpt 


Pott 


Title 

Accottplltbntnt 


.21 

.18 


* 

.34 


XnttrptrtOTul 
Rtlttlonthlpt 



^Corrtcttd tpllt htlf Tillabllltitt* 
^Tttt*,rtttit rtllablllxltt* 



Interscale and within-occadton correlations were nearly identical 
classroom situations correlated with nonclassiroom situations »32 for 
the pretraining and .34. for the post trailing , data* Again^ these 
intercPrrelations were higher than the intcirscale and between*^occasion 
intercorrelations which were *1S and ,21* * ^ 
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DESCRIPTIONS AND PSYCHOMETRIC DATA 
FOR CLIMATE SCALES USED IN THE 
EVALUATION OF RESEARCH UTILIZING 
PROBIiEM SOlVim, imEBBlRSOSAL 
IHFLVEUCE AND GROUP PROCESS SKILLS 



ERIC ' -^^^ . 



INTRODUCTiON * ^ 

* Outcome evaluations of t^ree of the Instructional systems developed 
by the Improving Teaching Competencies Program (ITCP) have Involved the 
use^ of various measures of classroom climate* This appendix presents 
the sout^es of these measures^ a*brlef summary and evaluation of the 

psychometric dat# available on the Instruments and a. summary* of 

'* * 

psychometric data-obtalne4 from the Northwest Regional Educational 
Laboratory (NWREL) evaluation studlas* 

The Instructional systems being evaluated through the classroom 
climate measures Included Reaeardh Utilizing Problem Solving (RUPS), 
Interpersonal Influence (INF), and Group Process Skills (GPS), which Is ^ 
part of the Instructional system Preparing Eduoaticnat Training 
Consultants I CPETC-Ih While these Instructional systems are independent^ 
they all focus heavily upon Interpersonal skills and processes* That Is^ 
much of the training Is designed to focus explicit attention on-lntier* 
personaloprocesses and to heighten awareness of certain .ispects of 
Interpersonal relationships* All three systems are designed .to be 

appropriate for classro^'om teachers and are Intended to have some effects 

* ' ' - 

on their behavior* 

The climate measures used In evaluating these Instructional systems 
Included scales selected from four Instruments* the Student, Activities ^ 
Questionnaire* My Class inventory* Student Attitude and Activity Survey* 
and the Student Behavioral Description Questionnaire* 

4 

I \ 

Structure of the Appendlx -^ 

This appendix has been divided Into two sectlqns* The first 
section Includes a description of the climate scales and a brief summary 
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and evaluation of the published psychometric data available oil the 
instruments* The second sectioa includes reliability data in the fc^rm 
of intraclass correla.tions and test-'retest reliabilities as well as 
scale intercorrelations computed' from data collected in the IXCP 

J* 

evaluation studies. ^ , - , 



/ 
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DESCRIPTIONS OF SCALES AND PUBLISHED PSYCHOMETRIC INFORHATICW 

student Activities Questionnaire 

The Student Activities Questionnaire was constructed' for >the 
evaluation &f an ESEA Title III project. Project IMPLODE, which was hypp- 

. theslzed to^lmpact upon classroom climate* It was designed to emphasize 

* , \ 

the Impact of the classroom process rather than Its Input to the educa- 

-i 

tlonar system* That Is, to determine the trails or abilities of the 

Students* A description of the Item generation and piloting procedures 

~ — ^ - .. .A — — = — 

♦ ' 

.le presented In "TKe Measurement of Academlp Climate In Elementary 

Schools" (Ellison, Calltier, Fox and Taylor, 1973) » The questionnaire 

♦ 

contalos, sixty multlple^cholce It^ns and -eight scales* Five of the 

eight scales have been used 'for the ITCP evaluation work* One scale of 

the Student Activities Questlonaalre was dropped because It was designed 

*as an implemenf:atlon measure for Project IMPLODE* Hence, It was not 

expected to be relevant to BVBSy INF or GPS training* Two additional 

scales (Career Dev'elopment and Independent Development) .were Judged to 

be of low relevance to the Instructional systems developed by the ITCP* 

The scales which were used Included: 

Enjoyment of School ; A measure of students* enjoyment of class 
activities and school work 

Reinforcement of Self ^Concept ; A measure of the lamount of positive 
feedback received by students, either, through personal contact or 
structured class activities ^ - . 

Classroom Participation ; **'A measure of student participation In 
class activities — frequency of class discussions, number of students 
who typically participate and opportunities for participation 

Democratic Classroom Control ; A measure. of the amount of student 
^ Input Into classroom decision making, planning of Individual 

activities and enforcement of rules ^ ^ 

• 

Indlvlddallsatlon of Instruction ;- A measure of th^ extent that 
students perceive their teachers as sensitive to their own' 
Individual needs, progress and goals 
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Published psychometric data for the Student Activities ^Questionnaire 
consists of scale lntercorrelatlons» Intraclass correlation coefficients 
for each ,ltem and additional construct validity evidence In the form of 
treatment and comparison group differences. 

With a sample of 654 fifth and sixth grade students^ scale Intern; 
correlations of all 8 of the SAQ scales ranged from .14 to a .49» except 
for the multiple talent teaching and career development scales which 
contained some comnoD Items. (These two scales were not selected for 
the evaluation of ITCP systems.) ^ Of the five scales selected for use» 
^the Interscale correlations ranged from .14 to ^42* The mean Interscale^ 
correlation £br the five .selected scales was .26 as opposed^ to the mean 
Interscale correlation of .35 for the full set of 8 scales on the Stu- 
dent Activities Questionnaire. This Indicated greater scale Independ- : 
ence among the five scales used than among all eight of the scales* In 
other wcrds» the moTe redundant scales were not'used^. 

Item reliability Information In the form of Intraclass'correXatlon 
coefficients Is available on all of the quest^lonnalre Items. Of the ^< 
Intraclass correlations^ 33 were significant at the •Ol levels 8 were 
significant at the .05 lWel» and 18 were nonsignificant. Of the 5^ 
.scales selected^ 15 Intraclass Rs were significant at the .01 level* 5 

i 

were significant at the .05 level* and 9 were nonsignificant. The It^ems 
selected appeared to be neither .more nor less reliable than the complete 
set of 60 Student Activities Questionnaire Items. 

Additional construct validity evidence available for the Student 
Activities Questionnaire Is that mean comparisons between the experl* 
mental and control schools In the Project IMPLODE evaluation '^resulted 
In significant differences In the expected direction In all scales 
except Indlvlduallzaclon^of Inscrucclon. 
134 ' ^ \ 
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Student Attitude and Activity Survey (SAAS) 

The SAAS was developed as a part of a Utah ESEA» Title III Project^ 
the Utah System Approach to Individualized Learning (U-SAIU (Nelson/^ 
1973)* It was developed to assess outcomes of an affective nature as 
well as student perceptions of certain process considerations. Many of 

r I m 

the scales of the SAAS were developed to conceptually parallel the 
concepts measured with the Student Activities Questionnaire. There are 
two forms of the SAAS» a Primary Form appropriate for Grades 2 through 4» 



and an IntertaedlaCe Form Intendcd'for tld6 wicn tirades :> and b. There are 
17 scales Included In the SAAS-. Many of them» however^ were developed^as 
measures of implementation for the U-SAIL project and were not appropriate^ 
for evaluation of the three Instructional syst^s. " 

The scales which were used Include general climate^ reinforcement 
of self-Qoncept^ general school sentiment » use of process approach^ and 
participation In Individualized learning strategies. All of these scales 
came from Che Intermediate Form of the SAAS. 

Published reliability Information on the SAAS' Is limited to 
communalltles obtained In a factor analysis of the SAAS. variables. The 
reported communalltles range from '71 through .77. There was» however^ 
no reported reliability estimate for the use of process approach 
variable. o 

. 

My Class Inventory (MCI> 

The MCI was developed to conceptually parallel the Learning 
Envlrpnment Inventory for elementary level school children. The com- 

] 

plete^MCI includes 45 Items in 5 scales:, satisfaction^, friction^ 

competitiveness* dlffl^:ulty and coheslveness'. (the difficulty scale 

is not being used in the ^TCP evaluation work.) The scale reliabilities 
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of the MCI ranged from *34 through *77t based upoii an analysis of data 
from a sample of 655 subjects* There was no validity information reported 
in^^the manual for the MCI (Anderson, 1973), for it was still in develop- 
ment at the time it was selected for use in the evaluation of the ITCP 
training systems. 

Student Behavior Description Questionnaire (SBPQ) 

The SBOQ wad developed to assess the interpersonal needs of high 
school and junior high school students (Croft, 1966)* -Although the 
complete taps interpersonal variables in terms -or relationship's 

with parents, friends and teacliers, only the three scales meas^ring 
relationship with teacher factors were used in the evaluation of the 
three instructional systems of the ITCP* Student ^perceptions of rela- 
tionships with parents and friends are ;iot likefy linked to the training"- 
offered in RUPS^ GPS or INF. 

The SBOQ was developed primarily through factor analytic techniques* 
Thus, the scales are xelatively homogenous and independent* 
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PSYCHOMETRIC INFORMATION AVAILABLE FROM IMPROVING TEACHING 
COMPETENCIES PROGRAM 'EVALUAXION STUDIES * ■ . , 

Design Essentials 

, Psychometric evaluations com'^icted with data actua;ily used In an 
evaluation study are potentially more useful than published psychometric 
Information In analyzing technical limitations of the Instrument^- as 
used* Data from the 18 climate scales collected In the evaluatlona of 
RUPS^ INF and GPS were used for further psychometric evaluations* 

The psychometric Information presented In Table A Includes scale' 
reliabilities^ Intraclass correlations (Haggard^ 1958) and test^retest 
reliabilities as well as scale Intercorrelatlons* Since the evaluation 
designs for these studies Included pretralnlng and posttralnlng 
admljilstratlons of the climate scales^ there are two Intraclass ^ 
correlations for each climate scale as well as a test^retest reliability 
for each. climate scale* 

Data collected for these analyses- came from fourth^ fifths and 
sixth grade students In the classrooms of teachers ssslgned to one of ' 
three training groups (RUPS, INF and GPS)^ or a control group* Specific 
recruitment and sampling procedures are described earlier in this 
report and in the Inte'xVersmat Influence Field Test I^aat Study arid 
Expert Reviet^ (Blscox» Cutting and George^ 1976). Readers Interpreting 
Table A of Intercorrelationtf and reliabilities should be aware of three 
cautions: 

1* Tew teachers were randomly assigned to the four groups* 

However^ recruitment proced^ires were quite sltillar* Thus» the 
nonrandom assignment of teachers to groups is not expected &to 
have a major iiiipact on the reliabilities and scale 
Intercorrelatlons for the combined samples* 
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The sample sizes. In terms of teachers or classrooms, for these 
reliabilities and scale Intercorr^elatlons differ from scale to 
scale for two reasons: 

a* While students In the classrooms of teachers In the^-IJVF^ 
GFS and control groups responded to all of the cllaate 
scales treated In Table B, the students In the classrooms 
of teachers In the RVPS group responded to only ^Ive of the 
eighteen scales* (The five scales sre marked with an (a) 
In the table* X As a result the nuiDber of clasi^rooms 
associated with pret33t scores for the five scales marked 
with an (a) Is 84, while the number of classrooms associated 
with the other pretest scores Is 52* The number of clasa- 
rooms associated with posttest scores for the scales with 
an (a) Is 73 and the number of classrooms associated with 
the other posttest scores Is 44* Sample sizes for pretest 
and posttest data are Included In Table B* 

b* The original total sample size for these combined studies 

Involved 107 teachers, rather than the 84. teachers for whom 

. pretest data were available* There was 21 perdent missing 

or unusable data for the pretest scores and 32 percent 

missing or unusable data for pofttest sccaes. The spteclflc 

* 

Impact of these missing data ia not known* ^ 
The climate Inventories were administered differently In the 

studies* All of the students In the classrooms of RUFS-tva.ined 

ft 

teachers responded to the five scales from the SAQ* However, 
since data from 18 scales rather than Just 5 scales were needed 
for the classrooms of the INF^ GPS and control groups, different 
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Table B 



Number of Classrooms and Scudencs for Whom 
Climate Data Were Analyzed on WPS^ IHF^ 
GPS and Concrol Groups 





Pre 


" Post " • 


Scale 


Number of 
Classrooms 


Number of 
Stud'encd 


Number of 
Classrooms 


Number of 
Scudents 


Satlsfaccion 


52 


' 721 


44 


494 


Fricclon 


52 


721 


44 


494 


Compeclclveness , 


52 


721 


44. • 


494 


Coheslveness 


52 


721 


44 


494 


Enjoyment of School 


52 


721 " 


44 


494, 


RelnforcemenC of 
Self-Concepc^ 


84 


1499 


73 


,1213 

* 


Classroom Partlclpacloa^ 


'84 


1499 


73 . - 


— 1213 


Democratic Classroom 
Concrol^ 


84 


1499 

'I 


73 


1213 


'Indlvlduallzac Ion ' 
of Instruction^ 


84 


1499 


73 


1213 


'SAQ ToCal* 


84 


1499 


73, 


.1213 


Cllmace 




697 


. ■ 73 ' 


509 


Reinforcement of 
Self-Concepc 


52 


697 


44 


509 


General School 
Senclment 


52 


697 


44 - 


5tt9 


Process Approach 


52 


697 


44 


509 


Individualized 
Approach 


52 


. 697 


4A 


509 


Teacher Conslderaclon ' 


52 


697 


44 


509 


Teacher Thrusc 


52 


697 


44 


. 509 


Teacher Domination 


52 


697 . 


A4 


509 



^he RUFS sample responded only Co chese five scales. All ocher samples 
responded Co all scales given In this cable. 



^' lAl 

131 



administration procedures were required for. those^ three groups* 
The 18 climattt scales w^re divided Into two questionnaires » 
Forms A and B* The students In each of the classrooms of the 
INF^ GPS and control group teachers were then randomly assigned 
to two groups* Students In one of these groups (for each 
classroom) responded to Form A ancl students In the other group 
responded to Form B* Therefore^ classroom means for the RUPS^ 
teachers are based upon all students In each class* Classroom 
means for INF, GPS and the control teach ers are bas^d upon a 
random half of the students In each classroom* One of the 
jresults of this procedure Is that the Intraclass reliabilities ^ 
for the scales not used In the RUPS study are slightly lover 
than they would have been If all students In all groups had 
responded to all scales* 

Interpretation of Table A (Reliabilities) ^ 

The Interpretation of Table A Is limited here to an examination of 
the reliabilities presented* The Intraclass correlations for each scale, 
are presented along the major (larger) diagonal In Table A* , The Intra^ 
clasa correlation Is a measure of reliability based upon the ratio of 
between class variance minus within class variance to between class 
variance. The greater the agreement among students in the same classroom 
given consistent differences between classrooms^ the greater the intra-' 

t 

class correlation* The Intraclass correlation^ then» is a measure of 
relative agreement within predefined groups* It can be Interpreted as 
any reliability coefficient where true score is defined as differences ^ 
in classroom means and error Is defined a& within class variance* Since 
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the Intraclass correlation It based upon one test administration^ .there 
are^two Intraclass correlations Cor each tcale^ one Cor the pretest and 
one for the posttest. 

Tesfretest rellabllltled» In the minor (smaller)^ diagonal* were ' 
based upon a pretralnlng and posttralnlng administration oC the climate 
scales. They must be viewed the:; as conservative estimates of stability. 

My Glass Inventory (MCI) . Pretest IntraclaM correlations for the 
MCI sca^t ranged from .29 (p < .033) for competi tiveness to .58 (p < .001) 
£or coheslveness. 

Posttest Intraclats correlations ranged, from (17 (p < .180) for 
coheslveness to .64 (p < .001) for satisfaction. The most consistently 
reliable scale from the MCI was the satisfaction scale with Intraclats 
correlations of .S3 (p < .001) and .64 (p < .001). Test-retest 
reliabilities for the MCI scales were quite low» ranging from *.03 to '.25. 

Student Activities Questionnaire . Pretext liltraclass correlations 
Cor this questionnaire ranged from .08 (p < .317) for enjoyment of school 
to .79 (p < .001) for democratic. classroom control. Posttest Intraclass 
correlations ranged from .25 (p < .078) for enjoyment of school to .72 
(p < .001) for democratic classroom control. Recall^that the enjoyment 
of school Intraclats reliabilities are based upon fewer classrooms and 
fewer students per classroom than for the rest of the scales. Test* 
retest reliabilities raided from .07 for enjoyment of school to .59 for 
classroom participation. Clearly the enjoyment of school scale Is much 
less reliable than the rest of the scales. The two most reliable scales 
from this questionnaire were classroom participation and democratic 
classroom control. 

Student Attitude and Activity Survey (SAAS ). Pretest Intraclass 
correlations from the SAAS ranged from .42 (p < ^002) for both climate 
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and process approach to .71 (p < *.001) for Individualized approach. 
Posttest Intraclass correlations ranged from .41 (p < .004) for process 
approach to .64 (p < .001) for reinforcement of self-concapt and 

Individualized approach. Test-retest rellabllltlas ranged ftroti ^48 for 

- ■ >■ 

process approach to .74 for climate. Test-Pretest reliabilities for the 
^AAS scales were much higher than those for the MCI scales while they 
were based upon approximately the same number of students and classrooms. 
One design difference which may have been a factor^ however* was that 
the MCI was part of Foim'X and the SAAS was part of Form B. Thus» 
different students were 'the respondents for these two sets of scales. 

Student Behavior Description Questionnaire (SBDQ) . Pretest Intra- 
class correlations for the SBDg ranged from .53 (p < .001) for teacher 
domination to .74 (p < .001) for teacher consideration. Posttest 
Intraclass correlations ranged from .43 (p < .003) f or tjeacher domination 
to .d9 (p < .001) for teacher consideration. Test-retest reliabilities 
ranged from .48 for teacher thrust to .65 for teacher consideration. 
Test-retest reliabilities for jthe SBDQ scales were similar to those for 
the SAAS and much higher than those for the MCI. Again design differences^ 
specif Ically, Inclusion of the SBDQ and SAAS In Form B and the HCt In 
Form A » may account for the similarities and differences. 
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UNCORRELATED t-VALUES FOR RESEARCH 
UTILIZING PROBLEM SOLVING 
POSTTRAINING AND FOLLOWUP 
QUESTIONHAIRES 
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t*Values foe ftUPS Fosccraining and Followup QutfCionnaicts* 







Posttniotng 


Tollomip 








Topic 


. Vsridbles 


X 


SD 


X 


SD- 


t-Vslue 


df 


P*Valutt 


Satisfaction 


1 =■ 

2-32 


4.14 


.89 . 


3.68 


1.03 


,-2.06 


72 * 


,*05 




3-33 


■ 3.46 


1.10 


3.24 


.93 




72 


*26 




5-34 


3^68 


.88 


'3.51 


1.10 




72 


*31 




26-40 " 


2.19 


.i 


2.54 ' 


.84 


1*74 


71 


*27 




29^2 ^ 


2.05 


1.13 


,2.75 


.87 


2*96 


71 


*0X 




- 30-43 . 


2.03 


1.24 


2.27 


1.19 


*86 


"^72 


*28 


USft and ' 


8-35 


3.83 


.88 


3.43 


1.12 




71 


*21 ' 


Application 




















9-36 


3.95 


.88 


3.46 


1.10 


-2*10 


72 


*04 




10-37 


3.49 


.80 


3.57 \ 


1.07 


*36 


72- 


*37 

* 




11-38 


3.35 


' .82 


3.19 


.94 


-.78^ 


72 


.29 



^The scale for Itesa 26;^29 and 30 (40, 42 and 43) is rtversad* 



Appendix F*: 

RESPONSES TO THE FOIXOWUP 
QUESTIONNAIRE: OPEH-ENDED QUESTION 

t 
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Queeticn 32: Now that y^u have been away from the workehop for eom 

.time^ have you changed yotdr opinion of how ueefut it woe? 



■No 

No ■ - 

No, I feel same. 

No 

Yes* I got tired of coming here. But time ha« shown that I 
really use many of the techniques. 

No/ 1 still feel the RUES process is not applicable on a day to " 
day problem; but because of all the steps of RUPS would be only ^ 
- useful for major .problems and one which you -have a great amount 
of time to work with* 

^So 

No 

Yes, I feel negative' I think much could be consolidated and the 
use of manipulation could be incorporated. l*ve been to a 
communications workshop which presented many of the same ideas, 
but we got into different groups, had objects ,to work with (a - 
physical problem to Mork on). It made it more interesting and 
more b^rable. 

Ho, '^I always felt that the workshop was useful, 6ut I think I 
could have had a little more guidance through writing out my 
force field and;force field analysis^^for some Incorrect forces 
' that might have been stated in my force field and analysis. ^ 

Hoi 

Yes. In use it was often too cumbersome and time consuming. 
Some portions clearly emerged as being more useful than others. 

Ho * ■ , ' 

No; it's quite useful.'- 

No, I've felc from the beglnalag' ic would be useful and ic was. 
. No ■ 
No 

No, v^en I had a successful small RUPS class project, ic made me 
feel better. Also I can see how I could. Introduce a RUPS project 
again to my faculty. ' " ■ 
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Same feelings* 

Yes. I thought I wo uld_^gngre_£h.e concepts presented and not put 
tnem into practice, but I haven't. Good* 

After the weekend sessions I experienced much enthusiasm about 
Implementing my newly gained skills. 

No 

Nd 

IS 

No. I feel I*ve learned much, even though I migh£ not go through 
the full process recommended by the course. 

It was useful. 

No. I find it has made me more aware of classroom' problems, the 
feasibility of discussing it with my class and coworkers. I find 
then I'm not alone on a particular problem. 

No, but it is more time consuming than I had anticipated. 

No ' 

.1 * 
No. Personally, I'll use parts of RUPS process, not the whole 
thing. 
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PREFACE 

This 'publication is one of a series of st^nmary evaluation reports 
issued the Northwest Regional Educational Laboratory to clocuiDent 
evaluation findings for selected products* The subject of this report 
is ReaeoPch Utilizing Probtm Solving (RUPS)f an instructional syst^n 
developed in the Improving Teaching Competencies Program* 

This report summarizes the technical report Reaearah Utilizing 
Problem Solving: Outcome Evaluation Report which presents the data 
collected about the Impact of the system on the classrooms of teachers 
trained in two RVPS workshops. The information is intended to provide 
evidence related to the impact of RUPS training on students* Although 
this information is primarily summative iti nature^ it should also help 
those who may be considering modifying the system'^to increase the 
likelihood of achieving impact on students. 

An institutional technical review has been conducted by Laboratory 

specialists external to the Program* Qualified evaluation consultants 

? 

external to the Laboratory have also reviewed this report* 



Lawrence D* Fish 
Executive Director 
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INTRODUCTION 



The purpose of this report Is to suomarlze the Research Utilizing 
Prpbtem Solving: Outccm Evaluation Report^ prepared by the Improving 
Teaching Competencies Program (ITCF) of the Northwest Regional 
Educational Laboratory (NWREL)\ The full report Includes evaluative 
data on trainee attitudes toward the Research Utilizing Problem Solving 
(RUFS) Instructional system^ trainee attitudes toward themselves^ 
trainee behavior and classroom lippact/ The reader Is referred to the ^ 
full technical report for specific design details related to the 
samplings Instrumentation and data analysis procedures used In the study* 

The RUPS Instructional systeoLt composed of 16 cumulative and 
sequential' training unlts» was designed to Increase the skills bf 
educators In systematically carrying out a 5'Step' method of probl^ 
solving; (a) Identify the problem^ (b) dlagnose^the prQl>lem situation^ 
(c) consider alternative actions » (d) try out a plan of action and 
(e) afdapt the plan* * 

Participants are guided through a sequence of unlts^ ea^h consisting 
of a sj&rles of concept papers* group discussions and exercises* The 

*■ 

materials were designed to develop participant knowledge and participant 
ability to use the RUPS process In Identifying and diagnosing classroom** 
school* and peer*related problems* 

The Instructlonal^'strategy of the RUPS system Is based on a pattern 
of repeated diagnosis carried out In small training groups of three or 
six persons* The knowledge gained through this process ^provides the^^ 



^Murray* Stephen L*» Rachel L. Rasden and Stuart M* Speedle* Research 
Utilizing Problem Solving: Outcome Evaluation Report* Portland > 
Oregon; Northwest Regional Educational La4)oratory» .1976^ 154 pp. 
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bssis for selecting and designing sctlon plans to solve the Identified 
problems* A simulation exercise provides opportunities |or partlclpsnts 
to practice skills In training groups and to learn to observe snd 
Improve their teamwork behaviors In s workshop setting* 

The Instructlonsl system was designed for mass distribution and la 
available In both classroom and administrator versions* Materials 
Include a leader*s guide* participant materials, sn sudlotape recording 
and a test* The Instructional strst'egy calls for one 27%*'hour mrkshop 
and two 3-hour followup' sessions held 6 to 12 weeks after training* 

A limited Educational Resources Information Center sesrch was 
conducted In order to Identify problem*solvlng trslnlng packages attd 
Instructional systems that offered alternatives to EUVS* This limited 
search did not Identify any alternative to the Wt^ Instructional system^ 
In the form of a complete snd comprehensive prepackaged training program 
that emphasized both problem solving atul group dynamics* 
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TEST SITE AND SAMPLE- CHARACTERISTICS 

i 

i 

t 

' Two test 8ltes In the Pacific Northwest were used for the outcome 
evaluation of SUPS. All fourth-^ fifth* and slxth*grade teachers at 
two test areas were Invited to participate In the i?C/P£^ workshop. These 
teachers were offered fre^^ materials » training and paid tuition for 3 
hours of graduate credit In return for their (Cooperation In providing 
necessary evaluation Information* ' 

It was originally anticipated that one major workshop site (Tacoma^ 
Washington) would he used for the RUPS impact study. Recruitment was 
Initiated In the Tacoma school district aiKl In several school districts 
south of Tacooa, Because of a teacher strike^ the Tacoma school 
district decided to withdraw from participation In the study and thus 
only teachers from the school districts south of Tacoma were still 
available. As a result of^ this situation^ not enough teachers were 
left to complete the evaluation. Thereforje^ additional teachers were 
recruited from a secojjd major site In Seattle^ Washington. Arrangements 
were made to conduct workshops atj^oth sites. In Tacoma^ a workshop 
was conducted for 17 participants and In Bellevue^ Washington^ a second 
workshop was conducted for 21 participants. 

The otlglnal evaluation design called for a comparison group 
composed of teachers randomly selected from those teachers who volunteered 
for RUPS training. However^ the physical distance between the two work' 
shop sites and the relatively small number of participants at each site 
precluded the u^e of such a group. An ITCP evaluation study Involving 
two other experimental groups and a control group was scheduled for the 
Seattle area two weeks before the RUPS workshop. The two experimental 
groups were^ to receive training In two other Instructional systems 



developed by the^ITCP; Interperaonat Influence (WF) and Group Prcceee 
Skills (GPS), The third group was a nontreatment control group scheduled 
to receive INF training the following spring* 

Since recruitments procedures for the GPS, ISF and control groups 
were essentially the same as those for the RUPS groups (participants for 
the GPS, INF and control groups were recruited from all fourth^^ fifth*' 
and slxth-gra4e teachers In the Seattle area)» It was decided that the 
24 teachers who were members of the nontreatment control group could 
also serve as the comparison group for the ROPS evaluation* 

Analysis of a background questionnaire revealed that the RUPS group, 

and the nontreatment control group were similar In terms of proportion 

/ 

of males to females^ ages» years , of teaching experience and reasons for 
attending the workshop* However^ ^ somewhat higher proportion of non*- 
treatment control group members had obtained a master's degree and had 
attended two or more previous NWKEL workshops* The specific effects of, 
these pretralnlng group differences on the outcome measures used are 
.unknown* 
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TRAINEE ATTITUDES TOWARD THE SYSTEM^ ] 

^ I, 

Participants reported high levels of satisfaction with the content 
and structure of the training* Participant responses to specific 
questions related to content and structure In the posttralnlng queatloa- 
nalre ranged from 57 to 87 percent positive* Of the partlcjLpants, 73 
percent summed up their experience In the SUPS vorkshpp as positive; 
the same proportion Indicated they would recommend RUP$ to friends with 
similar Interests* It should b€t aoted, however, that ratings of the 
workshop were slightly lower when participants were asked' to make 
Judgments on the basis of personal criteria, such as the degree to 
which the workshop addressed Issues they thought were vital or the 
degree to which the workshop was successful In meeting personal 
expectations* Some of the questions on the followup questionnaire, 
which was administered 3 months after training, were Identical to 
questions asked In the posttralnlng questionnaire; It was, therefore, 
possible to compare responses to these questions through means of 
uncorrelated t-teats* ^The results Indicated that positive opinions on 
the content, structure and satisfaction with tlie workshop did not 
change over a 3-month period. Moreover, the tendency of participants 
to respond less positively when Judging^ the workshop on personal criteria 
did not change* Negative changes did occur 9n two Items: participant ' 
ratings of the success of the 'workshop In offering new Insights and the 
number of participants summing up their workshop experience as positive* 

Participants were questioned about the usefulness of the RUPS 
tralnkng and Its applicability (o their work* Immediately after the 
workshop, participants tended to view RUPS training' as both useful and 
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applicable to their work (S7 percent eald they were planning to use BVPS 
a$ an Integral part of their work).. Of the participants^ 79 percent eaw 
RUPS AS useful In work with etudentSt 49 percent saw RUPS m useful In 
work with Superiors, and 57 'percent saw RUPS as useful In work with other 
teachers* The clsssroooi version of RUPS was used In the training and 
it seems reasonable to conclude that participants found It mo^re useful 
in that setting* 

The Identical Items on the posttralnlng and followup questionnaires 
that dealt with the usefulness and applicability of the training were 
compared through the use of uncoVrelated t-tests. The results Indicated 
that trainees did not change their positive perceptions of the usefulness 
and applicability of BVPS training In relation to work with teachers » 
Superl9rs and others* However, there was a decrease In the perceived 
success of the workshop In providing Information, with 'practical applica- 
tions for work with jtudents* The responses to the other Items concerning 
the usefulness of the training remained positive over the 3*-month period* 
The few negative changes that occurred probably reflect a more realistic, 
understanding of the applicability of RUPS to the school environment In 
contrast the general enthusiasm generated by the workshop* However, 
positive attitudes reflected In both the posttralnlng and the followup 
questionnaires suggest that participants did not receive excesslvi) 
negative ^feedback In their attempts to employ BVPS techniques* 

Both the observations of^ the evaluators who were present at each 
workshop site and the results of a weekly log that was f<llled out by 
trainees on a weekly basis were used to assess the significant side 
effects resulting from use o^ the system* A high degree of participant 
enthusiasm for the training was observed and documented at both the 
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conclusion of the workshop and at the followup sessions* Additionally, 
an examination of the weekly logs shoved that: SUPS participants tended 
'to select for their Improvement proj ects r elat ivel y coqip lex problems 
^requiring commitments of at least 30 days for completion* However* 
because there were no comparative data for these log entries* it Is 
not possible to determine whether this effect is indeed due to RUPS 
training or Is a reactive. effect of the data-gathering Instrument. 



ERIC 



-rsr 



TRAINEE ATTITUDES TOWARD SELF 



Correlated t^tests were performed on selected Items 'from the post* 



training and followup questionnaires to determine whether the trainees' 
perceptions of how well they understood the concepts and their ability 
to apply RUPS skills changed during the 3-fflonth Interval* Responses to 
the posttralnlng questionnaire Indicated that a majority of participants 
felt they understood the materials but did not feel^'that they could 
apply ttie skills Imnedlately after training* Similar results were 
obtained on the followup questionnaire administered 3 months later* 
The only islgnlf leant change was a positive change In the trainees* 
working understanding of "spotting and analyzing major results of data 
collected*** Thus» the data Indicate that trainee perceptions of their 
working understanding of the workshop materials and concepts were very 
positive when assessed Immediately after the workshop, and generally did 
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not change over a '3*month period* 



It was hypothesized tbdt» as a result of the trainings participant 

In RUPS would report an Increased probability of their using problem- 

/ 

solving techniques in a variety of situations* This hypothesis was 
evaluated through pretralnlng and posttralnlng administration of, a 
self-report attitude questionnaire called the Problem Solving Ori^tatlon 
Questionnaire (PSOQ)* There was a statistically significant incifease 




the self-reported probabilities of engaging in task-r4lated classroom 



probl&n solving. - However, SUPS training apparently did .not affect 
trainee perceptions of their own orientations toward interpersonal 



aspects of problem solving or nonclassroom problem/solving */ 

■ ■ / ^ 

The effects of RUPS training upon problem^solving orl/entation 

/ 

were also evaluated through a path analysis techniqu 
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e applied toy'the 

/ 
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WPS group daca base and the control group data base. Results of the 
analysis failed to support the hypothesis that SUPS trainees were more 
inclined to engage in p rob lea-solving beiiavior than (lontrol 'group 
teachers* The only significant path coefficient was for age; therefore, 
younger teachers saw themselves as more inclinded to use problem-solving 
techniques in the classroom* ^ 
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TRAINEE BEHAVIOR 



Information compiled from the weekly logs was used to assess^ _ 

whether MJPS trainees discussed and used (heir problem- solving knowledge 
and skills In their lo^al schools* The RUPS participants reported 
engaging In a mean frequency of 18*13 RUPS-likef problem-solvlng 
activities per week with students and 19*36 pet week with faculty and 
colleagues* Apparently RUPS participants do use problem^solvlng sliCllls 
In their classroom work and with faculty and colleagues* However » since 
no comparative data were available for Interpreting these results^ 
either from other training groups or from RUPS participants before their 
trainings It cannot be determined whether these data reflect (a) training 
caused changes in problcm^solvlng behavior or (b) typical teacher per^ 
formance* When working with students » participants had a tendency to 
use and discuss dlagnostlcally and analytically oriented skills; when 
working with faculty and peers » they had a tendency to use skills that 
were more oriented toward generation of solutions* 




CLASSROOM IMPACT . - . . 

It was hypothesized that» since RUPS Is a workshop In teacher 
problem solving for use In classrooms^ participation In RUPS would lead 
to a change In the classroom problem-$olvlng behaviors of teachers and 
thus have an Impact on the classroom learning environment* Although no 
direct measure of tefcher behavior was taken ln*the study^ teacher 
attitude toward the application of problsm^solvlng techniques was 
'assessed through the PSOQ* Measures of the learning environment were 
based upon student responses to four scales' In the School Activities 
Questionnaire (SAQ)* 

Data gathered from pretralnlng and posttralnlng administration of 

. the SAQ were used to determine whether a significant change In the 

J' 

classroom learning environment of WPS participants had taken place* 
Student responses on the SAQ» analyzed through correlated t^^ests^ did 
""not change significantly^ except for a negative change on the scale 
pertaining to reinforcement of self-concept. 

Not only did' evaltiators exsmlne changes In the learning environment 
through correlated t-tests» they also used a path analysis technique ^ 
In order tro test a model that assumes the learning environment is 
determined by background variables (e.g.» age of teacher » ^ex of 
teacher and class slze)^ training in RVPS and teacher attitude toward 
the application of problem^solvlng technique* The background variables 
In this model were used strictly for control purposes^, and therefore 
their relationships to the learning environment do not have much 
evaluative utility* However^ the effect of WPS training on the learning 
environment has direct evaluative utility and the Impact of teacher 



attitude on application of problem^solvlng techniques had Indirect 
evaluative utility* In the. latter case the assumption Is t;hat teacher 
attitude toward application Is an expected- outcome of training, and, to 



the degree that It Is related to the learning environment. It la of 
Indirect evaluative utility: If training does have an impact on teacher 
attitude and the learning environment and If teacher attitude Is related 
to the learning environment, the analysis of teacher attitude In terms 
of Its Indirect evaluative utility Is Straightforward* When training 
does not have an impact on teacher attitude or the learning environment, 
even though teacher attitude, relates to the learning environment, the 
evaluative utility of findings are more dlf^flcuit to analyze* In this 
case, although it cannot be said that training was effective, it can^b^ 
said that relationships among the outcome variables do partially confirm 
the model used' to evaluate the tralnltig. 

The data failed to ^support the hypothesis that training affected 
either teacher attitude, as measured by the PSOQ, or the classroom 
learning environment, as measured by th9 SAQ* Since significant path 
coefficients linking teacher attitude to the learning environment (as 
measured by three SAQ scales) were produced, some support was found for 
the hypothesis that teacher orientation toward problem solving is 
related to student perception of the classroom learning environment* 
The three scales used were; classroom participation, individualization 
of Instrucltlon and democratlc'classroom control* (The low rellablllty- 
of the fourth SAQ scale precluded its use in this analysis*) 

In the data collected for this study, there is no evidence of* 
behavioral changes that have an Impact on secondary audiences which may 
be_attrlbuted to RUPS training. Although the participants did see 

J 56 

14 



ERIC 



themselves as using and discussing their problem-solving skills with 
students and peers and did express a high degree of satisfaction with 
the training^ there is no evidence to support the conclusion that their 



postworkshop behav^U>r was different from* their preworkshop behavior or 
the conclusion that^ if their behavior was indeed different^ it had a 
measurable Impact on the secondary audiences in terms of the aspects 
of learning that were measured in this study^ 
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